🔍 Read the full analysis: Z.ai’s Legal AI Gets About 1 In 5 Benchmark Tasks Fully Right. Why The High Rank? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai says its GLM-5.3-FlashX-Legal model achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s archived leaderboard records 19.17%, placing it seventh of 81 models; the same evaluation reports a 92.5% pass rate across individual criteria. The ranking reflects a strict whole-task measure, not a claim that the model produces error-free legal work for general use.
Z.ai’s GLM-5.3-FlashX-Legal achieved an all-criteria pass rate of 19.17% on the Harvey Legal Agent Benchmark, placing it seventh among 81 models on Vals AI’s archived leaderboard. The result means the model satisfied every required rubric item on roughly one in five evaluated tasks—not that it earned only that share of available credit: Vals also reported a 92.5% pass rate on individual criteria.
The benchmark uses all-pass grading: a task counts as complete only if every criterion in its rubric is met. A missed deadline, citation, recommendation or other required detail can make the entire task a non-pass, even if the model handles the remaining items correctly. Under this rule, GLM Legal missed at least one criterion on about four in five tasks in the evaluation.
Vals AI’s archived Version 1 leaderboard lists the score as 19.17%, the model’s estimated cost as $0.15 per test, and the base GLM-5.3-Flash model at 6.67%. The difference is 12.50 percentage points, matching Z.ai’s stated improvement over its base model. The leaderboard says the GLM Legal criteria-pass rate was 92.5%; unlike the all-pass score, this measures individual rubric items, and Vals says an item counted only when both of its AI judges passed it.
Z.ai describes the system as designed for legal research, contract review and drafting, including initial drafts. Its documentation says the model received legal-domain mid-training and post-training, with reinforcement learning across tens of thousands of task environments modeled on legal workflows. The company recommends using it as an agent connected to services for laws, cases and company data. Z.ai’s October 13 blog post announced an application-based beta; its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens.
Why Whole-Task Scores Matter
The ranking offers a useful signal for organizations comparing legal AI systems, but the gap between 92.5% of criteria passed and 19.17% of tasks completed shows why the scoring method matters. Legal work products often contain multiple requirements. A document can be mostly sound and still fail a complete-task test because one important fact, citation, deadline or recommendation is wrong or absent.
For firms, the result does not establish that the model can safely handle client work without review. It instead suggests a system that may get many individual rubric items right while still falling short of every requirement on most benchmark tasks. Procurement decisions also involve questions the score does not answer, including how client information is handled, which services process it, and what verification is needed before generated work leaves a firm.
The low per-test cost reported by Vals may attract buyers evaluating models for controlled workflows. But the benchmark price is not necessarily the full cost of deployment: agent integrations, data controls, human review and correction can add expense. The leaderboard is one comparison point, not a complete assessment of reliability, privacy or legal suitability.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Amazon Kindle Scribe (32GB) with Premium Pen, Tungsten
Apple Watch Series 8 GPS, 41mm, Midnight Aluminum Case with Midnight Sport Band (S/M)
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas. Each task gives an agent an instruction and a client matter with relevant materials, then asks it to produce a work product. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting.
Harvey’s stated approach treats a task as complete only when every criterion passes. Vals AI runs a public leaderboard using Harvey’s grading protocol and says its private holdout set contains 120 tasks, with internet access disabled. Two language-model judges, GPT-5.5 and Claude Sonnet 4.6, grade submissions; the leaderboard score averages the task pass rates calculated by the two judges.
On the archived leaderboard, GLM Legal ranked below several higher-scoring systems, including Muse Spark 1.2 at 25.42%, Muse Spark 1.3 Max at 23.75% and Gemini 4 Argon at 19.58%. It ranked above the listed base GLM model. Harvey previously reported that the best all-pass rate among frontier models in its May 26 baseline was 7.1%, but that was a different run and is not a direct comparison with the Vals result.
“An all-criteria pass rate of 19.2%”
— Z.ai, in its model documentation
What the Score Cannot Establish
The leaderboard does not show how GLM Legal performs across the full range of live legal work, or whether its benchmark ranking predicts performance for a particular firm, jurisdiction or matter type. The holdout contains 120 tasks, and the score depends on the benchmark’s task mix, rubric design and AI-judge process. The available figures do not establish the model’s error rate for each type of legal task.
Z.ai calls the result state-of-the-art among open-source models, but its cited statement does not specify a comparison set. The source material also does not establish that publicly released weights are available for this legal model. Separately, the source’s author reports a four-task spot check in which the model cited rulings the author could not locate. That small check is not a benchmark result and cannot establish a general error rate; it does underline that the leaderboard score alone does not validate citations.
Details about data handling, the beta’s availability, and the checks users should apply to generated work are not fully specified in the cited materials. Z.ai’s pages describe outputs as drafts for legal professionals, but the source reports finding no “not legal advice” notice on the pages reviewed. That observation does not establish the terms or safeguards that apply to every product use.
Beta Access and Independent Checks
Z.ai’s October 13 announcement opened an application-based beta, and the company’s documentation describes use through agent connections to legal and company-data services. Further evidence will be needed to determine how the product performs outside the benchmark, what controls accompany those integrations, and whether Z.ai will publish more detail about its comparison set and model availability.
For buyers, the next practical step is to test the model against their own representative work, with qualified reviewers checking every factual statement and citation. Firms evaluating it will also need answers about client-data processing, retention, access controls and responsibility for review. New leaderboard entries or a later benchmark version could change the rankings; the current figures describe the specific Vals listing and evaluation, not a permanent assessment of the model.
Key Questions
What does the 19.17% score mean?
It is the share of benchmark tasks on which the model met every required rubric criterion, according to Vals AI’s archived leaderboard. A task with one or more failed criteria does not count as a complete pass.
How can the criteria score be 92.5% while the task score is 19.17%?
The measures use different units. The 92.5% figure concerns individual rubric items; the 19.17% figure concerns whole tasks, each of which requires every item to pass. A model can satisfy most criteria overall but still miss at least one on many tasks.
Is GLM-5.3-FlashX-Legal the top-ranked model?
No. Vals AI’s archived Version 1 listing places it seventh of 81 models. Z.ai’s “state-of-the-art among open-source models” wording refers to a comparison it does not define in the cited statement.
Does the benchmark show the model is safe to use without review?
No. The score is a benchmark result, not proof that outputs are reliable enough for unsupervised legal work. The source materials describe outputs as drafts for legal professionals, and firms would need to verify citations and other requirements before relying on them.
How does the model compare with its base version?
Vals AI lists GLM Legal at 19.17% and the base GLM-5.3-Flash model at 6.67%, a difference of 12.50 percentage points. That matches Z.ai’s stated improvement on this leaderboard comparison.
Source: ThorstenMeyerAI.com
