🔍 Read the full analysis: What Can LLMs Tell Us About The Novelty Of A Research Idea? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers tested six large language models as judges of scientific idea novelty and found that their results could change sharply with evaluation choices. In one pairwise test, small prompt changes shifted accuracy by more than 50 percentage points; retrieval and additional reasoning did not consistently help. The team released a benchmark and evaluation tools, while warning that some labels are imperfect.
Researchers at Ai2 report that six large language models gave sharply different judgments of the novelty of scientific ideas when researchers changed evaluation prompts and other setup choices. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points, raising questions about using a single LLM score to screen research proposals.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, along with Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The researchers assessed models using both absolute novelty ratings and pairwise comparisons. They also tested whether literature retrieval and giving models more reasoning would improve judgments; according to the authors, neither change helped consistently.
For one benchmark, the team mined ICLR 2026 peer reviews for explicit comments about originality. Its higher-novelty group contains 154 ideas from accepted papers in the top-rated 10% of their research areas, chosen when reviewers praised novelty and none disputed it. Its lower-novelty group contains 145 ideas from rejected papers in the bottom 10%, selected when reviewers raised originality concerns and none praised novelty. The study authors stress that acceptance or rejection alone is not a reliable novelty label.
A second test setting matched 154 human-authored ideas with 154 ideas from a basic language-model generator. The researchers treated the generated ideas as less novel, but said those labels were weak and not individually verified. In an expert check of 30 pairs, selected heavily from cases the strongest judge got wrong, the expert preferred the human idea in every pair and judged none of the generated ideas novel. The team also reported that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Unstable Scores Affect Research Screening
Research groups may use LLM judges to narrow a set of candidate ideas before committing time to literature review, experiments or expert assessment. If a judgment changes substantially after a small prompt edit, the resulting score may reflect evaluation choices as well as the idea. That makes scores harder to compare between projects or teams and could cause promising proposals to be discarded—or weak ones to pass an early screen.
The findings also bear on efforts to use AI to assist scientific discovery. Systems that generate hypotheses or synthesize papers need credible ways to assess whether a proposal adds something to existing work. This study does not show that every LLM novelty evaluation fails. It does show that performance on one selected setup is not enough to establish that a model can judge novelty reliably.
Products Worth Considering
Apple iPad Air (4th Generation, 10.9-inch, Wi-Fi, 256GB) – Space Gray
Apple iPad Air 11-inch (M2), 512GB, Wi-Fi, Space Gray
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
As an affiliate, we earn on qualifying purchases.
How Ai2 Built Its Novelty Tests
Ai2 frames the study as part of work on tools to support scientific discovery, including systems for exploring structured data, generating and testing hypotheses, and synthesizing research. A novelty judge is a model asked to assess whether an idea is new relative to prior work, either on its own or in comparison with another idea.
The authors say there is no standard recipe for building such judges: evaluations can vary by model, prompt, comparison format, reasoning budget and access to retrieved literature. They also argue that judges are often tested on human-written papers even when their intended use is to assess AI-generated ideas. The study therefore includes separate human-idea and human-plus-generated settings, while acknowledging that its reviewer-based labels are evidence about originality rather than a definitive measure of it.
“Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.”
— Ai2 study authors
Limits of the Novelty Labels
The benchmark’s labels have limits. Reviewer comments and paper rankings serve as signals of relative novelty, not a definitive or universal ground truth. In the human-plus-generated test, the researchers assumed that basic model-generated proposals were less novel than selected human ideas without verifying every proposal individually. The expert check covered just 30 pairs, not the full benchmark.
The supplied study description does not provide complete results for every model, prompt, reasoning budget or evaluation format. It also does not establish whether the pattern will hold across other disciplines, different idea generators or systems with broader literature access. The authors say some conclusions about disagreement between evaluation setups do not depend on the benchmark labels, but the description does not detail how broadly that finding applies.
Further Tests Across Research Fields
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to help researchers build additional evaluation sets. Other teams can use those resources to test whether the reported instability persists across fields, idea sources and judge designs, and to examine methods that might make judgments more consistent.
The release description does not announce a specific follow-up study or timeline. For now, the benchmark provides a way to compare evaluation choices; it does not identify a prompt, retrieval method or model design that can be relied on to produce dependable novelty judgments.
Key Questions
What did the Ai2 study find?
The study reports that six LLMs’ judgments of scientific idea novelty varied across evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The researchers tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.
Did literature retrieval or more reasoning improve the results?
Not consistently, according to the study authors. The supplied description does not give full results for every retrieval setup or reasoning budget.
Are the benchmark’s novelty labels definitive?
No. The human-idea labels draw on reviewer comments and paper rankings, which are signals rather than definitive measures. The generated ideas were treated as less novel without individual verification, and an expert review covered 30 pairs.
What did Ai2 release?
Ai2 released the study paper, benchmark data, code and an automated pipeline for building additional evaluation sets. The source description does not specify a release date or a planned follow-up timeline.
Primary source: Hugging Face · via ThorstenMeyerAI.com
