[Paper Review] The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Leo Gao, Stella Biderman+10 more|arXiv (Cornell University)|Dec 31, 2020

Topic ModelingComputer Science50 references483 citations

TL;DR

The Pile is a large, 825 GiB English text dataset composed of 22 diverse sources, designed to improve cross-domain generalization for language models. Models trained on the Pile outperform those trained on CC-100 and raw Common Crawl across multiple components.

ABSTRACT

Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present extit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction.

Motivation & Objective

• Motivate the need for diverse, high-quality training data beyond Common Crawl for broad-domain language modeling.
• Introduce the 22-component Pile and its construction, including new and existing datasets.
• Demonstrate improved downstream performance of models trained on the Pile compared to CC-100 and Raw CC.
• Provide extensive documentation and tools to enable reproducibility and informed dataset use.

Proposed method

• Assemble 22 diverse, high-quality datasets into a single 825 GiB English text corpus (The Pile).
• Duplicate- and quality-control steps include using Pile-CC and various preprocessing choices per component.
• Train 1.3B parameter models on different data sources to compare cross-domain generalization.
• Evaluate using bits per UTF-8 byte (bpb) as the primary metric, with per-document perplexity computed for GPT-2/GPT-3 across components.
• Perform size-controlled downsampling and deduplication to enable fair cross-dataset comparisons.
• Analyze dataset topics and document characteristics to document potential concerns and biases.

Experimental results

Research questions

• RQ1 Does increasing dataset diversity via the Pile improve cross-domain generalization for language models compared to training on CC-100 or Raw CC?
• RQ2 Which Pile components most impact performance, and how do models trained on the Pile fare on academic, programming, and multilingual content?
• RQ3 How does GPT-2/GPT-3 perform on individual Pile components, and what are the implications for future data composition?
• RQ4 What ethical and documentation considerations arise from training on such a large, diverse corpus, and how can they be addressed?
• RQ5 How does downsampling to fixed size affect conclusions about dataset quality and generalization?

Key findings

• Models trained on the Pile show significant improvements over CC-100 and Raw CC on nearly all Pile components when evaluated on held-out data.
• GPT-2/GPT-3 zero-shot perplexity on several components reveals that academic and domain-specific texts (e.g., PubMed Central, ArXiv, FreeLaw) are challenging without targeted data, highlighting the benefit of Pile diversity.
• Pile-CC alone provides limited gains; other components contribute substantially to cross-domain performance, especially for academic, programming, and mathematical content.
• The Pile components are not largely redundant with GPT-3 training data, indicating broad coverage beyond web-based text.
• Filtering Common Crawl via perplexity-based methods can reduce diversity, potentially harming performance on some domains, underscoring the need for careful data curation.