[Paper Review] CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Guillaume Wenzek, Marie-Anne Lachaux+5 more|arXiv (Cornell University)|Nov 1, 2019

Natural Language Processing Techniques14 references242 citations

TL;DR

CCNet presents an automatic, scalable pipeline to extract large, high-quality monolingual corpora from Common Crawl, with language identification and a novel monolingual filtering step based on domain similarity to Wikipedia.

ABSTRACT

Pre-training text representations have led to significant improvements in many areas of natural language processing. The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved. In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages. Our pipeline follows the data processing introduced in fastText (Mikolov et al., 2017; Grave et al., 2018), that deduplicates documents and identifies their language. We augment this pipeline with a filtering step to select documents that are close to high quality corpora like Wikipedia.

Motivation & Objective

• Motivate the need for high-quality, large-scale monolingual corpora for pre-training language representations across many languages, including low-resource ones.
• Describe an end-to-end preprocessing pipeline that preserves document structure and deduplicates content from Common Crawl.
• Introduce a monolingual filtering step using domain-specific language models to approximate high-quality sources like Wikipedia.
• Provide scalability details and resource estimates for processing Common Crawl snapshots.
• Share datasets and tooling to reproduce and tailor the pipeline for different language targets.

Proposed method

• Process Common Crawl snapshots by grouping WET files into 5 GB shards and representing each web page as an entry in JSON.
• Deduplicate paragraphs within a shard using 64-bit SHA-1-based hashes to reduce boilerplate and non-content text.
• Identify language at the document level using the fastText language identifier with a 0.5 score threshold.
• Train language-models on targeted domains (e.g., Wikipedia) per language using SentencePiece tokenizers and a 5-gram KenLM model, then compute per-paragraph perplexity as a quality score.
• Split each language’s data into head/middle/tail terciles based on perplexity to approximate quality levels without discarding potentially useful content.
• Provide a reproducer tool that reconstructs outputs from a URL list for users without running the full pipeline.

Experimental results

Research questions

• RQ1 Can a scalable pipeline extract high-quality monolingual corpora from vast web crawl data across many languages, including low-resource ones?
• RQ2 Does an explicit quality filtering step based on similarity to high-quality domains (via language-model perplexity) improve downstream representation learning?
• RQ3 How does deduplication order (deduplicate before LID vs LID before deduplication) affect language coverage, especially for low-resource languages?
• RQ4 What are the resource and performance trade-offs (time, RAM) for processing large Common Crawl snapshots?
• RQ5 Do corpora filtered by the proposed method improve downstream language model performance (e.g., BERT-like models) compared to Wikipedia or unfiltered data?

Key findings

• The pipeline yields 3.2 TB of compressed documents across 174 languages from a February 2019 Common Crawl snapshot.
• English alone accounts for about 700 million documents and 532 billion tokens after processing.
• Per-language perplexity-based filtering correlates with higher quality blocks (head) and improves downstream representations (e.g., faster/stronger fastText embeddings; BERT-BASE models trained on CCNet data outperform Wikipedia-based training on XNLI in multiple languages).
• Deduplicating before language identification significantly improves low-resource language document counts compared to the reverse order.
• Training BERT-BASE on CCNet data with comparable data volumes to Wikipedia yields performance gains on XNLI, notably a 7-point improvement for Urdu when using CCNet instead of Wikipedia.
• The approach demonstrates that CCNet enables scalable multilingual pretraining and improves quality over using Wikipedia alone for several languages.