🔍 Read the full analysis: FrontierCode: Exploring The Maintainer’s View Of AI Contributions on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository tasks judged against maintainer-style criteria. Claude Opus 5.5 led with a 52.2% score and passed all blocking checks in 57.7% of attempts; the results indicate that human review remains necessary, while model costs vary widely.
Artificial Analysis added results from FrontierCode to its evaluations on October 12, testing 23 AI models on coding tasks scored against maintainer-style review criteria. Claude Opus 5.5 topped the table at 52.2%, but cleared every blocking check in 57.7% of attempts, indicating that even the leading model did not consistently produce changes that met the benchmark’s acceptance conditions.
FrontierCode asks whether a patch would be acceptable in a real repository, not just whether it makes tests pass. Its creators, Cognition and open-source maintainers, designed tasks drawn from established projects. Artificial Analysis ran the models through the same open-source agent harness, Stirrup, allowing up to 250 turns per attempt and three attempts per task.
The benchmark checks patches on six axes, including hidden tests, regression and cleanliness commands, tests that must fail on the original code, adaptive checks for complex behavior, limits on files and diff size, and an LLM review against maintainers’ criteria. Any failed blocking check results in a zero; passing them earns a score from 70% to 100%, depending on quality checks. The source says nine of the benchmark’s 100 evaluated tasks were not solved by any model on any attempt.
GPT-6.1 Sol ranked second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Xiaomi’s MiMo-V2.6-Pro led the open-weight models listed, scoring 45.4%. Cost per task ranged from $0.09 for GPT-6 Luna to $10.15 for Claude Opus 5.5 among the stated cost-frontier models; Opus scored 3.1 percentage points above Sol at a reported 12.5 times its cost.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
The results put a measurable gap between working code and reviewable code. A change can pass ordinary tests yet still fail because it touches too many files, introduces regressions, or includes weak tests. FrontierCode’s scoring makes those concerns part of the result, reflecting the additional work maintainers face when reviewing AI-generated pull requests.
The leading model’s 57.7% blocking-check pass rate is not a measure of all real-world coding reliability, but it shows that success on this benchmark is far from consistent. Teams considering automated coding agents should not treat a model’s general intelligence score or test-passing ability as proof that its changes are ready to merge. The benchmark also highlights a practical trade-off: the top score costs substantially more per task than several near-leading alternatives.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Repository Changes
FrontierCode was announced in June 2026, with version 1.1 following in July and rules intended to prevent models from looking up task solutions online. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The full benchmark contains 150 tasks; Artificial Analysis used its 100-task Main set. The tasks are private, according to the source, to reduce the chance that models encountered them during training.
The scoring system also penalizes behaviors that can make patches harder to review, such as oversized diffs and edits outside relevant files. Attempts flagged by Cognition’s trajectory scanner for looking up solutions receive zero. Artificial Analysis used GPT-6 Luna (medium) for the code-review step. These design choices mean FrontierCode measures a specific combination of task completion, process constraints and judged patch quality, rather than coding ability in general.
Results also differ from general-model rankings. In the source’s comparison, Claude Sonnet 5.5 placed second among the listed models on Artificial Analysis’ Intelligence Index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. That contrast suggests general benchmark rankings may not predict repository acceptance. It does not establish how any model would perform across every software team or codebase.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The published results do not show how performance transfers to a company’s own repositories, review standards or workflows. FrontierCode covers a defined set of private tasks, and its LLM review component relies on GPT-6 Luna (medium); the source does not establish how different review judges might change scores. A benchmark score also does not directly measure the amount of human review a team would need in practice.
Several models had no result as of October 13, 2026, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence makes the comparison incomplete. The source also provides figures for selected models and costs, but does not establish that the reported per-task expenses will match usage in other settings. Results should be read as benchmark measurements, not guarantees of production performance.
As an affiliate, we earn on qualifying purchases.
More Model Results to Watch
The immediate next development is whether missing models receive FrontierCode results, particularly those the source identifies as having no score by October 13. That would broaden comparisons, though a new result would still reflect the benchmark’s task set and evaluation method.
For maintainers and engineering teams, the practical next step is to compare candidate models on representative work and review the resulting patches rather than relying on general leaderboards alone. The source does not announce a date for another FrontierCode release or say when additional results will appear. Until then, the published ranking offers a snapshot of 23 models on the 100-task Main set, with substantial differences in both score and reported cost.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether AI-generated changes meet repository-task requirements and maintainer-oriented checks, including correctness, regressions, scope and code review. Passing tests alone does not guarantee a positive score, because any failed blocking check results in zero for that attempt.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’ results, with a FrontierCode score of 52.2%. It passed every blocking check in 57.7% of attempts.
Does the result mean AI-generated code is ready to merge?
No. The benchmark is not a guarantee about code in any particular project. Its results show that the leading model did not pass all blocking checks on every attempt, so the benchmark does not support treating review as unnecessary.
How much did model costs vary?
The source reports costs from $0.09 per task for GPT-6 Luna to $12.00 for Claude Fable 5.1 across the models discussed. Among the listed cost-frontier models, Opus scored 52.2% at $10.15 per task, while GPT-6.1 Sol scored 49.1% at $0.81.
Which models were missing from the comparison?
As of October 13, 2026, the source listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
