TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A report dated Nov. 3 says 16 large language models attempted all 55 TIS-100 puzzles, finding 133 new points on the game’s performance trade-off frontiers across 18 puzzles. Opus 5.5 and GPT-6 Astra solved all 55; the models also set three outright cycle records, though players later improved on most of their programs.
A benchmark published Nov. 3 reports that 16 large language models completed Zachtronics’ programming game TIS-100, producing 133 new performance-frontier points across 18 of its 55 puzzles and setting three outright cycle records. The results suggest models can optimize tightly constrained programs, but the report also says human players improved on most of the models’ programs within days.
The benchmark authors gave each model the game and its manual, but withheld community code and exact leaderboard records. Each model attempted all 55 puzzles in one scored attempt, with a time allowance of 10 to 45 minutes per puzzle depending on difficulty. The set comprised the game’s 47 puzzles and eight community tournament puzzles. Models could read puzzle information, edit and run programs, and use diagnostic feedback about wrong outputs and deadlocks.
The report counts a new frontier point only when a program passed a fresh-game replay and 100,000 tests on an independent simulator. A frontier represents trade-offs among cycles, nodes and instructions, rather than a single score: a faster program may use more nodes or lines of code. The authors report 133 such points on 18 puzzles and three cycle records.
Opus 5.5 and GPT-6 Astra led on puzzle completion, solving all 55 and each finding new points on 11 puzzles. Opus produced 72 points to GPT-6 Astra’s 32; GPT-6 Astra had a slightly higher record-proximity score, 78.0 to 77.2. The report says the models’ proximity results do not establish a clear winner: the 95% bootstrap interval for their gap included zero. It also says GPT-6 Astra led in all six tested hypervolume settings, a separate measure of the trade-off space covered.
What the Scores Reveal About Coding
TIS-100 is a constrained test of writing and refining programs, not simply generating code once. Its processors have limited memory and short programs, and neighboring nodes must coordinate to exchange values. A solution must both return correct results and manage timing and resource trade-offs. The benchmark therefore offers evidence about how models handle iterative debugging and optimization under fixed limits.
The results also caution against treating puzzle completion as equivalent to optimization skill. Gemini 3.8 Flash solved 53 puzzles but found new frontier points on one; Fable 5.1 solved 52 and found new points on five. The distinction matters for readers comparing model performance: passing tasks, improving existing records and balancing competing constraints are different outcomes.
The findings do not establish that these models will outperform human programmers on general software work. They concern a specific game, tool setup, test suite and time budget. The follow-up from players is part of the story: model-generated programs provided useful starting points, but many were not the best available for long.
Products Worth Considering
Anker SOLIX C300 DC Portable Power Station with 60W Solar Panel, 288Wh, 300W
Samsung Galaxy Tab S7 11-inch Android Tablet 128GB Wi‑Fi with S Pen Mystic Black
Samsung Galaxy Tab A9+ 11-Inch Android Tablet, 64GB, Graphite
As an affiliate, we earn on qualifying purchases.
A Game Built Around Trade-Offs
In TIS-100, players program a grid of small processing nodes using a restricted instruction set. Each node has two registers and a limit of 15 lines, while communication between neighboring nodes can involve waiting. Even basic arithmetic can require coordinating several pieces of code. The game has built a community of players who compare solutions on a public leaderboard.
The benchmark’s measure of a new point reflects that leaderboard’s multi-objective character. A program can improve one dimension while worsening another, so neither the shortest program nor the one with the fewest cycles is automatically best across every measure. The authors say models saw coarse score histograms, but not exact records or community solutions.
One example in the report is the tournament puzzle Undecimal Twin Rounder, which rounds an input to the nearest multiple of N on one output and of 11−N on another. The community record at the time of the benchmark freeze took 2,694 cycles. Opus 5.5 first found a passing program after 6.5 minutes, then reached 1,918 cycles by 16 minutes into its run. That was 29% fewer cycles than the frozen record, but its solution used one additional node and 28 more instructions.
““Within days, players picked up the models’ programs and improved on most of them.””
— The TIS-100 benchmark report
Limits of the Benchmark Evidence
The report describes the results of a particular benchmark run; it does not show whether the same rankings would hold with different time limits, tools, model versions or puzzle sets. The source gives rankings and several aggregate measures, but does not provide enough information here to assess every model’s performance across all individual puzzles.
It is also unclear how much the findings generalize beyond TIS-100. The game’s small instruction set and explicit feedback make it unusually amenable to repeated testing. The 100,000-test check provides evidence that the counted solutions work beyond the game’s small built-in test set, but it does not establish performance on general-purpose software tasks.
The report says players improved on most model programs within days, but does not quantify the improvements or identify how many records remained after that follow-up. Its statement of three outright cycle records refers to the leaderboard freeze used for the study; the status of those records after player revisions is not specified in the supplied report.
Player Revisions and Reproducibility
The immediate next development is continued comparison with human solutions: the report says players began revising model programs within days. Whether the 133 frontier points or three cycle records persisted after those revisions is not stated, so the benchmark’s frozen results should not be read as a current leaderboard snapshot.
The authors say the game environments and model traces are available to research teams upon request. Access to those materials could allow other researchers to inspect how models used diagnostics, compare solution strategies and test whether the results can be reproduced. No further release date or follow-up study is specified in the report.
Key Questions
What did the LLMs do in TIS-100?
Sixteen models attempted all 55 puzzles in Zachtronics’ programming game, using the game and its manual but not community code or exact leaderboard records.
What does a new frontier point mean?
It is a program that adds a new trade-off among cycles, nodes and instructions. The benchmark required a fresh-game replay and 100,000 passing tests for a point to count.
Which models led the benchmark?
Opus 5.5 and GPT-6 Astra both solved all 55 puzzles and found new points on 11. Opus recorded more points, while GPT-6 Astra scored slightly higher on record proximity; the report says that gap was not statistically distinguishable.
Did the models beat human players?
The report records three outright cycle records against the leaderboard at the benchmark freeze. It also says players improved on most model programs within days, and does not give the updated record status.
Can the results be generalized to ordinary programming?
The benchmark tests performance in one constrained game with repeated runs and diagnostic feedback. The results do not establish how the models perform on general-purpose software development.
Source: tis-100.wincon.dev
