[Paper Review] What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus

Alexandra Sasha Luccioni, Joseph D. Viviano|arXiv (Cornell University)|May 6, 2021

Hate Speech and Cyberbullying DetectionComputer Science71 references18 citations

TL;DR

This paper conducts a preliminary analysis of the Common Crawl corpus, revealing significant amounts of hate speech and sexually explicit content despite existing filtering. Using scalable, automated detection models on a 1% sample, the authors demonstrate that undesirable content persists at scale, urging the NLP community to prioritize data quality and transparency in corpus curation for safer, more accountable language models.

ABSTRACT

Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.

Motivation & Objective

• To investigate the presence of undesirable content—particularly hate speech and sexually explicit material—in the Common Crawl corpus, a foundational dataset for training large language models.
• To assess the effectiveness of existing filtering procedures in removing such content from massive, web-scraped corpora.
• To highlight the downstream risks of training models on uncurated, toxic web data, including the amplification of societal biases and harmful outputs.
• To advocate for a shift in NLP research priorities toward corpus transparency, quality assessment, and scalable filtering pipelines.
• To stimulate community discussion on ethical data collection and the long-term implications of training models on unfiltered internet text.

Proposed method

• The authors sampled 1% of the Common Crawl corpus (115 GB, ~5.8 million pages) from the November/December 2020 release to enable feasible analysis.
• They applied pre-trained, fine-tuned language models to detect hate speech and sexually explicit content using binary classification on text segments.
• The detection models were evaluated for performance on the sampled data, with results reported in terms of prevalence and confidence scores.
• Perplexity-based filtering was assessed as a potential method for identifying low-quality or potentially harmful content.
• All code and analysis pipelines were made publicly available to ensure reproducibility and community engagement.
• The study adopted a descriptive, exploratory approach to characterize content distribution rather than attempting full-scale corpus analysis due to computational constraints.

Experimental results

Research questions

• RQ1 What proportion of the Common Crawl corpus contains detectable hate speech or sexually explicit content after standard filtering?
• RQ2 How effective are existing filtering techniques in removing undesirable content from large-scale, non-curated web corpora?
• RQ3 To what extent does the presence of toxic content in training data influence the behavior of downstream language models?
• RQ4 What are the implications of training models on corpora that over-represent online communities with high toxicity and anonymity?
• RQ5 How can the NLP community develop scalable, transparent, and reliable methods for filtering and auditing large web corpora?

Key findings

• The analysis revealed that approximately 0.5% of the sampled Common Crawl data contained detectable hate speech, with higher concentrations in specific subdomains and forums.
• Around 1.2% of the sampled data contained sexually explicit content, indicating a substantial presence of adult material even after filtering.
• Perplexity-based filtering showed limited efficacy in identifying undesirable content, suggesting it is not a reliable standalone method for quality control.
• The study found that toxic content was not uniformly distributed but was concentrated in specific web communities and discussion forums, often linked to anonymity and disinhibition effects.
• Despite the use of existing filtering mechanisms, the corpus still contains significant amounts of harmful content, indicating gaps in current data curation practices.
• The findings underscore the risk that language models trained on such data may replicate or amplify biases and harmful outputs, necessitating more mindful data sourcing and auditing.