[Paper Review] ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset

DatologyAI, :+33 more|arXiv (Cornell University)|Feb 16, 2026

Data Quality and Management0 citations

TL;DR

The paper shows that language-specific data curation improves multilingual model performance and compute efficiency, achieving a new Pareto frontier with a 20T-token curated pretraining corpus and strong multilingual results using far less compute.

ABSTRACT

Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.

Motivation & Objective

• Demonstrate that data quality and bespoke language-specific curation improve cross-lingual transfer in multilingual models.
• Show that English data quality benefits non-English languages and vice versa, challenging the notion of a strict capacity limit.
• Illustrate how multilingual curation affects compute efficiency and scaling at both 60B-token and frontier-scale pretraining.
• Provide a practical, language-aware curation framework that scales to 20T-token pretraining and beyond.

Proposed method

• Conduct controlled bilingual experiments with 3B-parameter models trained on 60B tokens to isolate data quality effects.
• Develop language-specific curation pipelines for 13 non-English languages plus English, including filtering, embedding-based selection, and synthetic data generation.
• Evaluate using multilingual benchmarks (Multilingual MMLU, Multilingual ARC, Belebele) and English counterparts with zero-shot, multiple-choice scoring, and cloze formulations.
• Assemble a 20T-token curated pretraining corpus deriving entirely from public sources and train 3B and 8B models with a 1T-token general multilingual mix.
• Implement a three-phase data curriculum with increasing multilingual token density to study scalability and Pareto efficiency.
• Compare against uncurated baselines and translate-augmented baselines to quantify the impact of data quality and translation.

Experimental results

Research questions

• RQ1 Does improving the English data quality improve non-English performance in multilingual models?
• RQ2 Does improving non-English data quality improve English performance?
• RQ3 Is bespoke per-language curation necessary to achieve optimal multilingual performance beyond English-centric strategies?
• RQ4 How does translation quality affect multilingual gains, and can it match bespoke curation when source data is high quality?
• RQ5 Can multilingual data curation shift the performance–compute Pareto frontier at large-scale pretraining?

Key findings

• Improving English data quality yields a 3.91% average relative improvement in non-English performance across 13 languages.
• Improving non-English data quality yields a 1.21% average relative improvement on English evaluations.
• Bespoke per-language curation yields up to 16.87% relative improvement over uncurated baselines for 3B models trained on 60B tokens.
• Translation benefits depend on source data quality; translating high-quality, score-filtered English data yields larger gains than random translations.
• A 1T-token training budget with about 8% multilingual data (~80B tokens across 13 languages) achieves strong multilingual performance, improving compute efficiency by 4–10× over strong baselines.
• DatologyAI models pretrained on 20T-token curated data define a new Pareto frontier, achieving competitive multilingual accuracy with far less compute compared to open-weight baselines.
• Multilingual curation scales to frontier models (e.g., Trinity Large) and yields exceptionally strong multilingual performance relative to training FLOPs.
• Language similarity to English correlates with transfer gains from English curation, with closer languages benefiting more.