🔍 Read the full analysis: GPT-6 Astra’s Review Shows What Green Tests Missed In Claude’s Edit on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
GPT-6 Astra, running in Codex, reviewed a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5. The reviewer found faults that passed unit tests, including stale results being reused and weight changes occurring at zero edit strength. The 10 published benchmark tasks showed more answered questions after editing, but the legal score remained zero.
A review by OpenAI’s GPT-6 Astra, running in Codex, found reproducible faults in a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5. The findings included a command-line failure missed by green unit tests and changes to model weights even when the edit strength was set to zero, raising questions about how the experiment’s results should be interpreted.
The experiment’s roles were split after two AI sessions began working on the same project on November 4, 2026. Claude handled the model edit, benchmark setup and blind grading; GPT-6 Astra was assigned to review the implementation. According to the source article, the reviewer did not alter the main test rig, inspect hidden benchmark tasks or run a competing model-improvement experiment. Claude’s session made the reported fixes.
The first review identified four reproducible problems. A resumed run could reuse answers from a different model or sampling setup; an unknown task ID could finish as a successful run with no evaluations; the actual command line failed despite passing parser unit tests; and converting weights back to a compact format changed values even at zero edit strength. The author says fixes were made for each, including rejecting unknown IDs and storing edited tensors in higher-precision bf16 rather than re-quantising them.
The updated article says the experiment covered 10 published Lexassay tasks. The edited model answered all 10, compared with four for the unedited model, while its legal score remained zero. Those outcomes are benchmark results reported by the source, not evidence that the model gained legal competence. The public runs are available on Lexassay, according to the article.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Test Provenance Matters
The review highlights a problem that can be easy to miss when a benchmark report focuses on its final score: a test suite can pass while the real program behaves incorrectly. In this case, a run could appear complete without evaluating any questions, or reuse answers produced under different settings. Either issue could make a result look more reliable than the underlying process warrants.
Weight conversion also complicates interpretation. In the first review round, the source reports that a zero-strength conversion changed 105,315 of 1,048,576 reconstructed values, or 0.84% of the matrix. In a later review measurement, storing one real matrix in a different number format moved it by 0.51% and changed 915,360 values. These are measurements of numerical conversion, not legal performance or model error rates. They show why the editing operation must be separated from changes introduced by saving or converting weights.
The account also describes corrections to the interpretation of results, not only repairs to code. The author says the reviewer challenged claims that the zero-strength result was “exactly identical,” that a per-answer average of wrong citations needed no qualification, and that one control supported a strong causal conclusion. For readers evaluating AI research, that distinction matters: reproducible runs and careful wording affect how much a reported result can support.
Products Worth Considering
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, IPX4, Black
Apple Watch SE 3 GPS 40mm Smartwatch
Amazon Kindle Scribe (3rd Generation), 64GB, 11-inch Display, Graphite, with Premium Pen
As an affiliate, we earn on qualifying purchases.
How the Review Was Set Up
The underlying experiment attempted a directional weight ablation on Ornith 1.5, inspired by a MIT CSAIL post about weight-space programming. According to the source, Claude’s session identified a “loop direction” in the model and weakened it in the weights without training. The benchmark comparison—10 answered tasks after editing against four before—does not show an improvement in the legal score, which remained zero.
GPT-6 Astra’s role was deliberately narrower than running a second experiment. Its report described the assignment as: “Review and validate the other session’s implementation.” It used separate check programs and small reproducible examples, while the Claude session retained control of the main implementation and applied the fixes. The article says the reviewer did not examine hidden prompts, topics, answers or reference solutions.
The source reports that, in the second review round, all 12 tests written by the author passed, while four of the reviewer’s six provenance checks failed. That is a result from this project’s review process, not a general estimate of how often AI tests miss errors. The article says the runs and related materials are public on Lexassay.
“The rig’s own measurement reported an intended change of zero.”
— Thorsten Meyer, describing the zero-strength conversion measurement
Limits of the Reported Checks
The source article describes a review of one experiment, and it does not establish how common these faults are in other AI research projects. It also says GPT-6 Astra did not re-run every test during its final documentation step, was not a fourth grader, and measured conversion on a CPU rather than the production GPU path. The article does not provide enough information here to independently assess whether those differences would alter the measurements.
The benchmark comparison is limited to the 10 published tasks described in the update. The legal score stayed at zero, and the source does not establish that the edit improved legal reasoning beyond the reported task-answer count. It also reports two separate conversion measurements from different review stages; those figures should not be treated as a single estimate or as a model-quality metric. The precise contribution of the directional edit, as distinct from other changes, remains uncertain.
What the Public Runs Can Show
The next step for readers seeking to assess the claims is to inspect the published Lexassay runs, including the weight-edited model, edit control and model-card sampling materials cited by the source. Those records may help clarify the benchmark setup and the effect of the fixes, although the article does not say that an external replication has been completed.
The source presents the review as an account of an experiment and its checks, not as legal advice or a general validation of weight-space editing. Further confidence would depend on reproducible evaluation with clear controls, documented conversion effects and results across additional tasks. No follow-up date or independent replication is specified in the supplied material.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and measurements for a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5. The source says GPT-6 Astra did not run a competing model-improvement experiment.
What faults did the review find?
The first review found that resumed runs could reuse results from different settings, unknown task IDs could produce empty runs reported as successful, the real command line failed despite green parser tests, and weight conversion changed values at zero edit strength.
Did the edit improve Ornith 1.5’s legal score?
No improvement in the legal score is reported. The edited model answered all 10 published tasks, compared with four before editing, but the legal score remained zero.
Were hidden benchmark tasks inspected?
The source article says GPT-6 Astra’s second review stated that it did not inspect hidden benchmark prompts, topics, answers or reference solutions.
Can the reported results be independently checked?
The source says the runs are public on Lexassay, including the weight-edited run, an edit control and model-card sampling. The supplied material does not report a completed independent replication.
Source: ThorstenMeyerAI.com
