We recently benchmarked the new legal model from z.ai, called GLM Legal.
Our benchmarks give models whole pieces of lawyer work, start to finish. Does the model understand the question being asked? Does it read the documents? Can it reason over them? And does it then draft a coherent response that answers the question?
We ran GLM Legal on a range of tasks across several benchmarks, in a very spaghetti-on-the-wall sort of fashion. The work covered commercial and IP agreements, corporate and M&A matters, financing, real estate and land use, financial services, privacy, insurance, antitrust, civil litigation and personal injury, criminal appeals, public-sector employment, and tax. So basically, most of the major practice areas.
Here is what we ran.
Revise contracts. 126 assignments.
The public leaderboard, which includes GLM-5.3
Work five matters through 15 assignments, one after another.
Sol 6.1, and the public leaderboard
Draft a transfer-pricing Master File from 72 source documents.
GLM-5.3, z.ai’s bigger general-purpose model
Advise a simulated client, and ask for the records it needs.
Sol 6, Grok 4.7, Luna
Mark up a contract, explain why, and advise the client. 60 to 65 documents per matter.
Luna, Sol 6.1, Astra
View table
| Test | What the model had to do | Up against |
|---|---|---|
| ContractBench (public) | Revise contracts. 126 assignments. | The public leaderboard, which includes GLM-5.3 |
| LegalReasonBench (public) | Work five matters through 15 assignments, one after another. | Sol 6.1, and the public leaderboard |
| Legora BAR tax case (public) | Draft a transfer-pricing Master File from 72 source documents. | GLM-5.3, z.ai’s bigger general-purpose model |
| Our firm sim (internal) | Advise a simulated client, and ask for the records it needs. | Sol 6, Grok 4.7, Luna |
| The Inferno (internal) | Mark up a contract, explain why, and advise the client. 60 to 65 documents per matter. | Luna, Sol 6.1, Astra |
| Test | What the model had to do | Up against |
|---|---|---|
| ContractBench (public) | Revise contracts. 126 assignments. | The public leaderboard, which includes GLM-5.3 |
| LegalReasonBench (public) | Work five matters through 15 assignments, one after another. | Sol 6.1, and the public leaderboard |
| Legora BAR tax case (public) | Draft a transfer-pricing Master File from 72 source documents. | GLM-5.3, z.ai’s bigger general-purpose model |
| Our firm sim (internal) | Advise a simulated client, and ask for the records it needs. | Sol 6, Grok 4.7, Luna |
| The Inferno (internal) | Mark up a contract, explain why, and advise the client. 60 to 65 documents per matter. | Luna, Sol 6.1, Astra |
Every assignment comes with a grading checklist: the things a careful lawyer would have to get right. A score is the share of that checklist the answer earned. One mistake can cost several items on the list, so we show you the mistakes themselves as well as the numbers.
What we found: like most models, GLM Legal makes mistakes when it already has everything it needs to avoid them. In a settlement review it identified the correct costs, then used the wrong amount when it calculated the client’s recovery. In a lease review its own calculation produced May 30, 2017, and all three documents it handed back said May 1. In a custody agreement one paragraph deleted an addendum and another said the addendum continued.
This is the #1 failure mode we see across all runs, including from the best frontier models. A good harness (the tools and instructions wrapped around a model) can help, but not always. Since we want to test the models, and not the harness, we use a vanilla harness. It gives them seven basic tools and a turn limit. z.ai recommends running GLM Legal with its own legal research and Word tools. We did not.
ContractBench
ContractBench is a public benchmark of contract revision. Its 126 assignments cover 210 graded items.
GLM Legal scored 56.74%.
On the publisher’s leaderboard, Grok 4.7 is on top at 58.3%, GLM-5.3, z.ai’s bigger general-purpose model, sits at 55.0%, and Astra is at the bottom at 45.6%.
Contract revision scores
Our run Public subset
Score among valid grades.
Publisher Full benchmark
Our result covers the public task set. These publisher reference scores cover the full benchmark, including additional tasks. They are not a same-set model ranking.
View table
| Model | Score | Result scope |
|---|---|---|
| GLM Legal | 56.74% | Our run · valid grades |
| GLM Legal | 55.66% | Our run · failed targets included |
| Grok 4.7 | 58.3% | Publisher · full benchmark |
| GLM-5.3 | 55.0% | Publisher · full benchmark |
| Astra | 45.6% | Publisher · full benchmark |
| Grok 4.7 | 58.3% | Publisher · public subset |
| GLM-5.3 | 54.8% | Publisher · public subset |
| Astra | 45.8% | Publisher · public subset |
| Model | Score | Result scope |
|---|---|---|
| GLM Legal | 56.74% | Our run · valid grades |
| GLM Legal | 55.66% | Our run · failed targets included |
| Grok 4.7 | 58.3% | Publisher · full benchmark |
| GLM-5.3 | 55.0% | Publisher · full benchmark |
| Astra | 45.6% | Publisher · full benchmark |
| Grok 4.7 | 58.3% | Publisher · public subset |
| GLM-5.3 | 54.8% | Publisher · public subset |
| Astra | 45.8% | Publisher · public subset |
The pattern was edits that fixed one clause and broke another. In a services agreement, GLM Legal changed a termination right so that “either” party could terminate when Client breached. The trigger was still Client’s breach. So Client got an exit for its own breach. Not great, Bob.
In another answer, it required handing over the information needed to move testing to a replacement laboratory, and kept confidentiality language that could prohibit disclosing the methodology. In a third, it narrowed an indemnity and left a broader provision sitting beside it that could bring the same exposure right back. Check each change on its own and it looks fine. Read the contract as a whole and it doesn’t.
LegalReasonBench
LegalReasonBench is a public benchmark where each matter is worked through a series of assignments, one after another. The full benchmark has 50 matters. We only have access to five of them, 15 assignments in total. We emailed the vendor about the rest and have not heard back. Sad.
We ran GLM Legal through it twice. It scored 28.43% and 31.47%. Sol 6.1 scored 32.34%. The publisher’s leaderboard, which covers all 50 matters, runs from 31.9% for Sol 6 to 50.8% for Gemini 4 Argon.
Legal reasoning scores
View table
| Model / run | Score |
|---|---|
| GLM Legal · run 1 | 28.43% |
| GLM Legal · run 2 | 31.47% |
| Sol 6.1 | 32.34% |
| Model / run | Score |
|---|---|
| GLM Legal · run 1 | 28.43% |
| GLM Legal · run 2 | 31.47% |
| Sol 6.1 | 32.34% |
In the criminal appeal, both GLM Legal runs recited de novo review and then argued abuse of discretion. One of them also read the facts in the light most favorable to the verdict. Sol did a better job separating a refusal to give a jury instruction from the cases where an instruction had been given, and it used the incoming-fire facts in the record more carefully.
One GLM Legal answer took the civil clearance rate from the 19th Circuit’s row of a table. The question was about the 18th Circuit, the row next door. It fixed that in the next assignment. Both GLM Legal land-use answers named the wrong appellate district for the date the task supplied. Sol named the right one.
The answer key has mistakes of its own. One item docked answers over a nonretroactivity statement in a bylaw that does not contain one. Others confused the numbering of the assignments with the section numbers inside the answer. Those are on the key, not on the model.
Legora BAR: the public tax case
Legora’s public tax case is one big assignment: take 72 source documents and draft a company’s 2023 transfer-pricing Master File.
We ran it for this article with two models: GLM Legal and GLM-5.3. Sol 6.1 graded both reports against Legora’s rubric.
GLM Legal scored 53.7%. GLM-5.3 scored 53.1%. That is a tie.
GLM Legal vs GLM-5.3
between the reports
One public tax case, one answer per model. Both reports were regraded by Sol 6.1 with high reasoning. These are unweighted checklist scores, not Legora’s weighted BAR leaderboard scores. Difference: 0.6 percentage points.
View table
| Model | Score |
|---|---|
| GLM Legal | 53.7% |
| GLM-5.3 | 53.1% |
| Model | Score |
|---|---|
| GLM Legal | 53.7% |
| GLM-5.3 | 53.1% |
Our internal firm sim benchmark
The next two are ours. First, our internal super double secret, must-sign-an-NDA-that-cost-6.2h-to-review benchmark. Darn, cat’s out of the bag.
The firm sim puts the model in the lawyer’s chair with a simulated client. We run each matter two ways. With an incomplete file, the model has to ask the client for the records it needs. With a complete file, it gets everything up front. Either way it has to make a decision and advise the client. A list of issues to look into does not count.
The comparison covers two matters: a personal injury settlement and a licensing dispute.
Compare the file conditions
The model has to act like a diligent attorney
View table
| Matter and file | GLM Legal | Sol 6 | Grok 4.7 | Luna |
|---|---|---|---|---|
| Personal injury settlement, incomplete file | 73% | 86% | 77% | 46% |
| Personal injury settlement, complete file | 64% | 86% | 80% | 41% |
| Licensing dispute, incomplete file | 21% | 74% | 48% | 25% |
| Licensing dispute, complete file | 51% | 70% | 63% | 17% |
| Matter and file | GLM Legal | Sol 6 | Grok 4.7 | Luna |
|---|---|---|---|---|
| Personal injury settlement, incomplete file | 73% | 86% | 77% | 46% |
| Personal injury settlement, complete file | 64% | 86% | 80% | 41% |
| Licensing dispute, incomplete file | 21% | 74% | 48% | 25% |
| Licensing dispute, complete file | 51% | 70% | 63% | 17% |
The clearest error came in the settlement review with the complete file in hand. GLM Legal’s memo said: “True costs are $4,398.40, not $5,865.90.” Its tables and its note to the client then deducted about $5,866 anyway. That left the client’s stated net recovery $1,467.50 too low. Luna did the same thing in one run: it flagged the possible duplicate charge in its memo, then used the full $5,865.90 in the client’s numbers. All three Sol 6 answers and all three Grok 4.7 answers used the corrected costs.
The settlement cost error
“True costs are $4,398.40, not $5,865.90.”
The excess deduction understates the client’s recovery by $1,467.50.
View table
| Item | Amount |
|---|---|
| Correct costs identified in the memo | $4,398.40 |
| Costs used in the client calculation | $5,865.90 |
| Client recovery understated by | $1,467.50 |
| Item | Amount |
|---|---|
| Correct costs identified in the memo | $4,398.40 |
| Costs used in the client calculation | $5,865.90 |
| Client recovery understated by | $1,467.50 |
The same GLM Legal answer got real things right. It rejected a release for a defendant who was not paying, and it rejected a confidentiality restriction. Then its client note stated a less-than-75% rule and correctly worked out the threshold at $112,500. It told the client that a verdict between $112,500 and $150,000 would trigger the same fee consequence. That is above the threshold it had just calculated. Our checklist does not cover that point, so it cost the model nothing.
GLM Legal also does not follow up. In the five graded runs where it had to ask the client for records, it asked once in four of them and not at all in the fifth, an SEC subpoena matter. It never asked a second time. In the settlement matter, its one message asked for surgery estimates, among other things. The client answered the first three groups of questions, sent four records and skipped the rest. GLM Legal did not ask again, and later used the wrong scar-revision figures. In some runs it checked its inbox and picked up records the client had sent late. Checking the inbox is not the same as asking for what is still missing.
The Inferno
The Inferno is our internal benchmark of hard contract matters. Each one comes with 60 to 65 source documents that do not all agree with each other. The model has to work out which terms control, calculate what follows from them, and hand back three things: a contract markup, the analysis behind it, and advice the client can use.
GLM Legal ran three Inferno matters:
Digital asset custody. A fee dispute and operating controls under a custody agreement.
Index license amendment. Amend an index license and calculate the fees.
Commercial lease. A lease and the amendments that came after it.
The Inferno results
View table
| Matter | GLM Legal | Luna | Sol 6.1 | Astra |
|---|---|---|---|---|
| Digital asset custody | 8.3% | 11.1% | 38.9% | 36.1% |
| Index license amendment | 12.8% | 10.3% | 43.6% | 51.3% |
| Commercial lease | 5.6% | 0% | 36.1% | 41.7% |
| All three | 9.0% | 7.2% | 39.6% | 43.2% |
| Matter | GLM Legal | Luna | Sol 6.1 | Astra |
|---|---|---|---|---|
| Digital asset custody | 8.3% | 11.1% | 38.9% | 36.1% |
| Index license amendment | 12.8% | 10.3% | 43.6% | 51.3% |
| Commercial lease | 5.6% | 0% | 36.1% | 41.7% |
| All three | 9.0% | 7.2% | 39.6% | 43.2% |
Nobody aced these. Sol 6.1 and Astra got through far more of each matter than GLM Legal did, and neither got through all of it. Fable 5.1 took the same three during development in a different setup: every document in a single prompt, and no tools. It scored 30.6% on custody, 50% on the lease and 28.2% on the index license. Opus 5.5 scored 51.3% on an earlier version of the index license matter.
Where it went wrong, one example per matter:
Index license: the signed copy was right there. The file included the signed agreements, with the actual rates in them. GLM Legal never opened them. It searched the public filing version, where the figures are redacted, and treated the terms as unavailable. It also took a junior’s note at its word over the signed contract.
Digital asset custody: half the fix. GLM Legal spotted the right fee tier and wrote the right pricing source into its markup. Then its worked example changed the rate and kept the invoice’s price.
Commercial lease: third time lucky, sort of. Its first two attempts produced no documents at all. On the third, it used a tool to add 23 months to June 30, 2015. The tool said May 30, 2017. All three of its final documents said May 1, 2017. It also argued that certain rights had expired, on a theory a signed amendment had already superseded.
The lease date error
in the final documents
May 30, 2017
The date produced by adding 23 months.Select either marked date to inspect it.
The amendment used the earlier of the 23-month date and actual business occupancy. Without an earlier occupancy event, the delivered date starts rent 29 days early.
View table
| Step | Date / value |
|---|---|
| Start date | June 30, 2015 |
| Months added | 23 |
| Tool calculation | May 30, 2017 |
| All three final documents | May 1, 2017 |
| Difference | 29 days |
| Step | Date / value |
|---|---|
| Start date | June 30, 2015 |
| Months added | 23 |
| Tool calculation | May 30, 2017 |
| All three final documents | May 1, 2017 |
| Difference | 29 days |
Sol 6.1 and Astra got the license fee right, made both custody adjustments, and did better at following the lease amendments.
Odds and ends
A few other internal tasks, with no scoreboard.
An unfinished financing package. GLM Legal and Qwen both solved the round price. GLM Legal caught a Common-share conversion reserve that Qwen left out. But its workbook showed $4 million / $16 million where the “Value at P” columns needed $24 million / $36 million, and the audit cells still said PASS. The checks were not looking at the totals on display. The run hit a 60-request cap before it finished, so it has no overall grade.
Four cross-practice tasks. GLM Legal was better at recognizing an existing merger remedy and a partial-disclosure allegation. Both GLM Legal and DeepSeek repeated some of the defects in the analysis they were handed.
The investment adviser custody rule. DeepSeek and Qwen3.8-27B both held on to the condition that the assets must be client funds or securities. GLM Legal quoted that condition and then applied custody across the board. Qwen scored 80%. GLM Legal passed 87.3% of its checklist items, failed 9.1% and left 3.6% undecided.
A guaranty. Sol and Astra did better at connecting the releases, the survival clause and the waivers.
A guided collateral task. GLM Legal earned 100% and DeepSeek earned 90%, with the same grader. GLM Legal’s answer was the more complete one.
What it did well
It was not all carnage.
Contract revision. Its 56.74% on ContractBench is ahead of GLM-5.3’s 55.0% on the publisher’s leaderboard, and within two points of the leader.
Keeping up with Sol. On LegalReasonBench its better run, 31.47%, landed within a point of Sol 6.1.
The personal injury settlement. With an incomplete file it scored 73%, close to Grok 4.7 and far ahead of Luna. It rejected a release for a defendant who was not paying, and it rejected a confidentiality restriction.
Spotting the issue. In the custody matter it found the right fee tier and the right pricing source. In the financing package it caught a share reserve that Qwen left out.
The guided collateral task. It earned 100%, with the more complete answer.
What we took from it
GLM Legal can spot the issue, write pages of sensible analysis, and still hand the client the wrong number. Four mistakes cost it the most:
It did not open the signed document it had been given.
It followed an outdated amendment, or a junior’s note, over the document that controls.
It stated a rule and then applied a different one.
It got the analysis right and then put something else in the calculation and the final documents.
On the three Inferno matters, Sol 6.1 and Astra got much further on working out which document controls and on the math. In the firm sim, Sol 6 and Grok 4.7 used the corrected settlement costs every time. Elsewhere it was mixed, and GLM Legal did useful work on the financing, cross-practice and collateral tasks. On the one head-to-head we ran, GLM Legal tied GLM-5.3, z.ai’s bigger general-purpose model.
z.ai says it trained GLM Legal to read the file, check documents against each other, and produce a first draft. Contract revision is where it did that best. Keeping documents consistent with each other, its own included, is where it fell down.
Next we want to test four things. Does the model go and get the missing record? Does it pick the amendment that controls? Does it update every input in a calculation? And does the note to the client agree with the memo and the markup? Whether it finishes at all inside the limits is a separate test.