[Paper Review] EngGPT2: Sovereign, Efficient and Open Intelligence

G. Ciarfaglia, A. Rosanova+14 more|arXiv (Cornell University)|Mar 17, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

EngGPT2-16B-A3B is a sparse Mixture-of-Experts LLM designed for European sovereignty and efficiency, achieving competitive benchmarks with far less training and inference compute than larger dense models.

ABSTRACT

EngGPT2-16B-A3B is the latest iteration of Engineering Group's Italian LLM and it's built to be a Sovereign, Efficient and Open model. EngGPT2 is trained on 2.5 trillion tokens - less than Qwen3's 36T or Llama3's 15T - and delivers performance on key benchmarks, including MMLU-Pro, GSM8K, IFEval and HumanEval, comparable to dense models in the 8B-16B range, while requiring one-fifth to half of the inference power, and between one-tenth to one-sixth of the training data and consequent needed training power. Designed as a trained-from-scratch Mixture-of-Experts (MoE) architecture, EngGPT2 features 16 billion parameters with 3 billion active per inference, with expert sizes positioned between those used in GPT-OSS and Qwen3. Approximately 25% of its training corpus consists of Italian-language data, to deliver strong capabilities for European and Italian NLP tasks among models of similar scale. This efficiency aims to position EngGPT2 as a key contributor to the growing portfolio of open-weight European models, combining performance and efficiency with full alignment to the EU AI Act. EngGPT2 is also a single model capable of multiple reasoning modes: non-reasoning, reasoning in Italian or English, and turbo-reasoning (a concise, bullet-point style reasoning available in both languages designed for real-time reasoning use cases). EngGPT2 aims to set a new standard for resource-conscious, high-performance LLMs tailored to European and Italian contexts.

Motivation & Objective

• Promote a sovereign, open European AI ecosystem with strong regulatory alignment (EU AI Act).
• Develop an efficient, open-weight LLM architecture tailored to European and Italian NLP needs.
• Demonstrate that sparse MoE design can deliver competitive performance at a fraction of training and inference costs.
• Provide multilingual capabilities with substantial Italian-language data integration.
• Detail the training pipeline, evaluation methodology, and deployment considerations for enterprise and public-sector use.

Proposed method

• Propose a 16B-parameter sparse MoE transformer (EngGPT2-16B-A3B) with 64 experts per layer and 8 active per token.
• Use Grouped Query Attention (GQA) and SwiGLU activations with Rotary Positional Embeddings and RMSNorm for stability.
• Train in four phases (pre-training, long-context adaptation, mid-training, post-training) on up to 2.5T tokens across 128-node HPC GPU clusters.
• Adopt a Megatron-based distributed training stack with expert-parallelism and long-context support up to 32768 tokens.
• Extend the tokenizer (131084 vocab) with specialized tokens for reasoning and tool-calling; integrate compliant data licensing and copyright filtering.
• Evaluate using lm-evaluation-harness and BFCL EvalScope, with standardized and optimal serving configurations; report normalized efficiency metrics.

Experimental results

Research questions

• RQ1 Can a sparsely activated MoE model at 16B parameters achieve competitive performance against larger dense baselines when accounting for training and inference efficiency?
• RQ2 How does EngGPT2 perform on multilingual and Italian-specific tasks, including long-context reasoning and tool integration, relative to comparable baselines?
• RQ3 What are the impacts of extensive context windows and structured reasoning capabilities on downstream tasks such as code generation, math reasoning, and function calling?
• RQ4 To what extent can a European, open-weight model meet EU AI Act alignment and sovereignty objectives while maintaining enterprise-grade performance?

Key findings

• The model trains on approximately 2.5 trillion tokens across three pre-training phases and yields about 250,000 GPU hours for pre-training plus additional phases, totaling roughly 5.7e22 FLOPs.
• EngGPT2-16B-A3B employs 64 experts per layer with 8 activated per token across 24 layers, ensuring about 3B active parameters per forward pass.
• Average Model FLOPs Utilization (MFU) remains around 21%–31% during pre-training with long-context and mid-training phases showing lower MFU due to efficiency trade-offs, and post-training MFU staying below 10%.
• The model supports a 32768-token context window and utilizes a vocab size of 131,084 plus 12 specialized tokens for reasoning/tool-calling.
• On standard benchmarks, EngGPT2-16B-A3B achieves performance comparable to larger dense baselines when normalized for training or inference compute, while delivering lower training/inference costs.
• The training and evaluation framework emphasizes transparency, reproducibility, and regulatory alignment, including open publication on Hugging Face Hub and rigorous copyright/ licensing filtering.