🔍 Read the full analysis: Could 42 AI Setups Answer The Same Legal Questions? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer AI says its Lexassay test gave 42 AI setups the same 15 questions about German, US and EU law. Claude Opus 5.5 scored highest, while a model marketed for legal work scored 8 out of 100; the publisher says the results are a limited spot check, not a formal benchmark.
Thorsten Meyer AI has published results from a test in which 42 AI setups answered the same 15 legal questions, reporting that performance varied sharply and that errors included incorrect deadlines and citations. The test’s publisher, which runs Lexassay, says the results are a spot check rather than a formal benchmark and discloses that Claude agents graded the answers, with a Claude model ranking first.
Lexassay’s reported top scorer was Claude Opus 5.5, with 79 points, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leader fully met every criterion on only eight of the ten publicly available tasks, according to the published results. The report says that giving models more time to think did not improve scores.
GLM-5.3 FlashX Legal scored 8 out of 100 and received zero on the hidden set. Z.ai describes the model as optimized for professional legal workflows, according to the source article. It fully solved none of the 15 tasks and gave no answer on two, where it used its stated 64,000-token thinking budget. Its base model scored 3. The report says the legal model produced 21 real citations, 25 misattributed citations and one invented citation.
Mistral Large 4 scored 10. The article says it got about half the substance right but lost many points over citation errors. Across all published runs, the author reports that Lexassay checked 2,644 citations as real and identified 804 as invented or not supporting the claim made. The source does not provide a full model-by-model breakdown in the supplied material.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Errors Can Affect Legal Decisions
The test highlights a practical risk for people using chatbots to understand legal rights or act on a deadline: a plausible answer may still contain a consequential mistake. The article’s example involves two German civil cases with judgments served on April 27, 2027. Because Corpus Christi falls on May 27 and is a public holiday in Bavaria but not Berlin, the appeal deadline would move to May 28 in Munich while remaining May 27 in Berlin, according to the test’s account. One model reportedly gave Berlin the later date.
That distinction matters because, as the source explains, a late appeal in German civil procedure is rejected as inadmissible. The test also reports an answer that put the deadline for appeal grounds a month too late. These examples do not establish how AI performs across actual legal practice, but they show why users should not rely on an unchecked chatbot answer for deadlines, filings or legal claims.
The reported citation totals point to a second concern: a citation can name a real law or judgment while misrepresenting what it says. Readers checking an AI answer need to verify both that the source exists and that it supports the specific statement. Meyer advises using AI as a starting point, checking citations against the source, and consulting a lawyer where money, a deadline or an authority is involved.
Products Worth Considering
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
Samsung 34-Inch Odyssey G5 Ultra-Wide Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Lexassay Ran the Test
Lexassay used 15 questions covering German, US and EU law: ten published questions and five hidden ones. The source says the questions included deadline calculations, changes to law, major court decisions and prompts containing false assumptions. Models were tested without internet access or tools, so the exercise sought to measure answers from the models themselves rather than their ability to search supplied material.
Each answer was compared with a prepared answer using three to six weighted criteria. Citations were checked against official databases and classified as real, misattributed, invented or not findable. A misattributed or invented citation cost a quarter of the task’s points; answers without a legal-advice disclaimer or referral to a lawyer lost 5 percent, according to the method description.
The publisher says it tested different effort settings separately, which can cause a model to appear more than once among the 42 setups. It also states that each model answered each task once. That design means small score differences may reflect variation in individual answers, rather than stable differences in overall capability. A separate Vals AI legal-agent benchmark, cited by Meyer, ranks models differently; that test supplies documents to models, unlike Lexassay’s no-tools format.
“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”
— Thorsten Meyer, the Lexassay publisher
Limits of the Reported Scores
The results come from the operator of Lexassay, not an independent testing organization. Meyer discloses that his organization runs the test and that Claude agents graded the answers; a Claude model placed first. He says the method and public answers, gold answers and citation checks are published so readers can inspect the grading. The supplied material does not include independent replication of those results.
The one-answer-per-task design cannot show how consistently a model would respond to repeated versions of the same question or perform on a broader range of legal work. The test also excludes internet access and tools, so it does not directly measure legal systems that search documents or provide sources to a model. The available source does not list all 42 setups and their scores, or provide enough detail to independently assess every grading decision. Scores should not be read as a ranking of legal services or as proof that any model can safely handle a person’s case.
Check the Method and Citations
The source says Lexassay’s method page and published materials let readers examine the scoring, public answers, gold answers and citation checks; hidden-task scores are published without the answers. Readers can use those records to review how the test reached its conclusions. The supplied article does not announce a date for a new test or an independent audit.
For anyone considering AI for a legal issue, the immediate next step is to check any cited law or decision in an official source and to seek a qualified lawyer for a deadline, contract, claim or communication with an authority. The report itself is explicitly not legal advice, and its German deadline example should not be treated as guidance for an individual case.
Key Questions
What did the 42 AI setups answer?
They were given the same 15 questions about German, US and EU law, including deadline calculations, legal changes, court decisions and prompts based on false assumptions.
Which setup scored highest?
In the results reported by Lexassay, Claude Opus 5.5 scored 79, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher says the test is a spot check, and discloses that Claude agents graded the answers.
How did the legal-focused model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and zero on the hidden set. The source says it fully solved none of the 15 tasks and gave no answer on two.
Can these scores show which AI is best for legal work?
Not by themselves. Lexassay tested one answer per model and task without internet access or tools, and the publisher says small score differences may be noise. Another benchmark cited in the article ranks some of the same models differently.
Should I rely on a chatbot for a legal deadline?
No. The report describes deadline errors that could affect whether an appeal is accepted. Check the rules and cited sources with a lawyer; the article says it is not legal advice.
Source: ThorstenMeyerAI.com
