[Paper Review] Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

Julien Abadji, Pedro Ortiz Suárez+2 more|arXiv (Cornell University)|Jan 17, 2022

Web Data Mining and Analysis10 citations

TL;DR

This paper proposes a document-oriented, multilingual web corpus derived from OSCAR 21.09 by rearchitecting the Ungoliant pipeline to classify language at the document level instead of line level, preserving document integrity and enabling improved filtering through new automatic annotations. The key contribution is OSCAR 22.01, a 12GB multilingual corpus with deduplication, line filtering, and metadata annotations for quality-aware filtering, enhancing usability for NLP and digital humanities applications.

ABSTRACT

The need for raw large raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.

Motivation & Objective

• To address the limitations of line-level language classification in existing multilingual web corpora like OSCAR, which fragments documents and compromises readability.
• To improve data quality and usability for downstream NLP tasks by preserving document-level integrity during web crawling and processing.
• To reduce noise and improve filtering capabilities through automatic annotations such as document length, adult content, and deduplication markers.
• To enable more effective use of large-scale crawled corpora in training generative language models and supporting low-resource language applications.
• To mitigate biases and over-filtering in adult content detection, especially concerning LGBTQI+ content, by analyzing blocklist-based annotation limitations.

Proposed method

• Re-engineered the Ungoliant pipeline to classify language at the document level instead of line level, ensuring coherent document boundaries.
• Implemented line filtering to retain only contiguous lines above 100 characters, preserving document structure and readability.
• Added metadata annotations including document length (short, medium, long), adult content classification via URL blocklist, and deduplication markers.
• Applied deduplication at both document and line levels to reduce redundancy, especially in high-resource languages like English.
• Used a blocklist-based adult content filter to annotate potentially explicit content, with validation on French corpus samples.
• Evaluated the impact of hard thresholds (e.g., 5-line minimum) on document classification and quality perception.

Experimental results

Research questions

• RQ1 How does shifting language classification from line-level to document-level affect corpus quality and usability in NLP tasks?
• RQ2 To what extent do automatic annotations such as length and adult content filtering improve the discoverability and quality of usable documents in large crawled corpora?
• RQ3 What is the impact of hard thresholds (e.g., minimum line count) on document classification and perceived data quality?
• RQ4 How effective is a blocklist-based adult content annotation system in identifying explicit content without over-filtering non-explicit or LGBTQI+-related content?
• RQ5 Can deduplication and document-level processing significantly reduce redundancy while maintaining linguistic diversity in low-resource languages?

Key findings

• The new OSCAR 22.01 corpus preserves document integrity by filtering out short, fragmented lines, resulting in more coherent and human-readable documents.
• The corpus includes a 12GB multilingual dataset with deduplicated English content and a line deduplication tool, reducing redundancy.
• Approximately 3.2% of adult-labeled documents in the French corpus were found to be LGBTQI+-related, indicating potential over-filtering and representation bias in blocklist-based annotation.
• False positives in adult content detection were observed in 21 out of 100 sampled documents, including non-explicit educational content and websites previously used for adult content.
• Length-based annotations revealed that ~50% of documents in Czech were labeled as 'long', suggesting potential quality issues or line length discrepancies in mid-to-high resource languages.
• The pipeline identified that 30MB of Alemannic German content exists in the German corpus, suggesting opportunities for merging or treating regional varieties as distinct language resources.