📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project, built from scratch with extensive native-language data, outperforms multilingual models but scores near chance on Italian academic tests. This reveals the limits of current investment and scale in sovereign LLMs.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, despite outperforming comparable multilingual models on Italian benchmarks. This stark result underscores the significant challenge of scaling native-language models to handle complex academic tasks, even with substantial investment.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, trained models ranging from 350 million to 7 billion parameters. It was built on an open data and code foundation, with the largest model trained on 1.14 trillion Italian tokens, making it one of the most extensive native-language datasets used in European sovereign-LLM efforts.
Despite these efforts, Minerva-3B’s performance on the INVALSI benchmark reveals a critical gap: the model’s ability to handle academically rigorous language tasks remains near chance, even at high parameter counts and large data scales. Researchers have concluded that dataset size and parameters alone are insufficient to ensure proficiency in complex language understanding, especially in specialized domains like education.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Advanced Language Tool Kit: Teaching the Structure of the English Language
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Inspect AI: Writing Reproducible Evals and Safety Tests for LLM Systems
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Resurhang 200 Pcs Italian Flash Cards, Learn Italian Verb Conjugation, Present Tense Cards with English Translations and Example Sentences, Gift for Adults Beginners Travelers
200 Pcs Flash Cards: this flash cards set includes 200 flashcards focused on present tense verb conjugations, available…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Investment
This development challenges assumptions that larger native-language datasets and models automatically lead to deeper country-specific language understanding. It suggests that European efforts may need to significantly scale their investments beyond current levels to achieve meaningful country-knowledge depth. The results imply that strategic planning must incorporate the reality that current parameter scales may be inadequate for complex language tasks, even with extensive native-language data.
Background of Italy’s Sovereign-Language Model Efforts
Italy’s Minerva project represents a deliberate choice to train a large language model from scratch using a substantial native-language dataset, contrasting with approaches like Portugal’s AMÁLIA, which relies on continuation pre-training. The project was launched in 2024, supported by Italy’s National AI Strategy, and involved a team of 15 researchers and PhD students. It utilized Italy’s CINECA supercomputing infrastructure and aimed to demonstrate the feasibility of sovereign-language models in Europe.
Previous European projects, such as AMÁLIA, layered small amounts of European Portuguese onto multilingual foundations, whereas Minerva prioritized native-language training with a focus on Italian. While Minerva has achieved impressive benchmarks in some areas, its performance on academic content reveals significant limitations, raising questions about the effectiveness of current scaling strategies.
“Despite our extensive native-language training, the model still performs near chance on the INVALSI tests, showing that language understanding at this level demands more than just data and parameters.”
— Research team member
Unresolved Questions About Scale and Effectiveness
It remains unclear what specific architectural or training modifications could significantly improve performance on complex language tasks for sovereign models like Minerva. The exact threshold of data scale and parameter count necessary to achieve meaningful country-specific language understanding is still unknown. Additionally, the implications for future investments and strategic planning in European AI infrastructure are still being debated, as the current results suggest a need for reevaluation of assumptions.
Next Steps in Sovereign-Language Model Development
The Minerva team is continuing to iterate on their methodology, with upcoming experiments planned to test different architectures and training regimes. Further evaluations on domain-specific benchmarks and real-world tasks are expected to clarify whether increased scale or alternative approaches can bridge the current performance gap. Policymakers and researchers will likely reassess investment strategies based on these emerging insights, emphasizing the need for larger native-language datasets and possibly new model architectures.
Key Questions
Why did Minerva score so low on the Italian academic tests?
The evaluation suggests that despite large-scale native-language training, the model lacks sufficient depth in handling complex academic language tasks, indicating that dataset size and parameters alone are insufficient without architectural or domain-specific tuning.
Does this mean sovereign European LLMs are not viable?
Not necessarily. It highlights the need for more investment and possibly different approaches, but current results do not rule out the eventual viability of sovereign LLMs for specialized tasks.
What implications does this have for future AI investments in Europe?
The findings suggest that European AI efforts may need to significantly scale their native-language data and model sizes or adopt new architectures to achieve meaningful country-specific language understanding.
How does Minerva compare to other multilingual models?
Minerva outperforms comparable multilingual models on Italian benchmarks but still struggles with complex academic tasks, indicating that native-language training alone may not suffice without additional scaling or architectural innovations.
What is the significance of this research for broader AI development?
This research underscores that scale and native-language data are critical but not solely sufficient for high-level language understanding, prompting a reevaluation of strategies in both European and global AI initiatives.
Source: ThorstenMeyerAI.com
