[Paper Review] Into the LAIONs Den: Investigating Hate in Multimodal Datasets

Abeba Birhane, Vinay Uday Prabhu+3 more|arXiv (Cornell University)|Nov 6, 2023

Hate Speech and Cyberbullying Detection16 citations

TL;DR

The paper audits two open-source vision-language datasets, LAION-400M and LAION-2B-en, and shows that hateful content increases with dataset scale, and that image-based NSFW filtering does not fully remove toxic alt-text.

ABSTRACT

'Scale the model, scale the data, scale the compute' is the reigning sentiment in the world of generative AI today. While the impact of model scaling has been extensively studied, we are only beginning to scratch the surface of data scaling and its consequences. This is especially of critical importance in the context of vision-language datasets such as LAION. These datasets are continually growing in size and are built based on large-scale internet dumps such as the Common Crawl, which is known to have numerous drawbacks ranging from quality, legality, and content. The datasets then serve as the backbone for large generative models, contributing to the operationalization and perpetuation of harmful societal and historical biases and stereotypes. In this paper, we investigate the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. Our results show that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively using a metric that we term as Hate Content Rate (HCR). We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text. Instead, we found that trace amounts of hateful, targeted, and aggressive text remain even when carrying out conservative filtering. We end with a reflection and a discussion of the significance of our results for dataset curation and usage in the AI community. Code and the meta-data assets curated in this paper are publicly available at https://github.com/vinayprabhu/hate_scaling. Content warning: This paper contains examples of hateful text that might be disturbing, distressing, and/or offensive.

Motivation & Objective

• Motivate and ground the need for auditing large multimodal datasets beyond model-centric scaling beliefs.
• Assess how scaling from 400M to 2B-en samples affects hateful, targeted, and aggressive alt-text content.
• Evaluate the relationship between image NSFW labels and the accompanying alt-text toxicity.
• Propose methodological and policy recommendations for transparent, equitable dataset curation and usage.

Proposed method

• Sub-sample 100,000 image rows from each dataset shard to obtain 3.2 million (400M) and 12.8 million (2B-en) image-text pairs.
• Use pysentimiento hate-speech analyzer to obtain three scores per alt-text: hateful, targeted, and aggressive.
• Define Hate Content Rate (HCR) as the percentage of samples with a score exceeding a threshold P_threshold for each category and for Any-of-the-three.
• Compare HCR across datasets using threshold-based curves and Wilson score intervals to assess statistical differences.
• Perform file-wise HCR analysis across 32 shards (400M) and 128 shards (2B-en) and conduct Welch t-tests to compare means.
• Correlate NSFW image labels with alt-text toxicity using Pearson correlation on LAION-2B-en samples.

Figure 1 : HCR curves for the LAION400M and LAION-2B-en datasets using pysentimiento outputs showing that Hate Content Rate increased with dataset size.

Experimental results

Research questions

• RQ1 Does scaling LAION-400M to LAION-2B-en increase the prevalence of hateful, targeted, and aggressive alt-text descriptions?
• RQ2 How well do image-based NSFW filters align with the toxicity detected in alt-text descriptions?
• RQ3 Are file-level HCRs consistent with dataset-level HCRs when comparing shards across datasets?
• RQ4 What recommendations emerge for transparent, robust auditing and curation of large vision-language datasets?

Key findings

• Hate Content Rate (Any-of-the-three) is higher in LAION-2B-en than in LAION-400M across thresholds, indicating increased hateful content with scale.
• At P_threshold = 0.5, HCR for hateful speech reaches up to 0.7 in LAION-2B-en versus up to 0.6 in LAION-400M.
• File-wise HCRs are higher for LAION-2B-en across hateful, targeted, and aggressive categories, with strong statistical support (Welch t-tests, very small p-values).
• There is a slight correlation between NSFW image labels and hateful/targeted alt-text (correlation ~0.227–0.215), but weaker for aggressive content (0.076).
• NSFW filtering based on images alone does not reliably remove hateful or targeted alt-text; some toxic content remains in supposedly safe subsets.

Figure 2 : Fused swarm-box-violinplot that captures the file-wise HCR metrics for all the 160 (=32+128) parquet files from LAION400M and LAION-2B-en. HCRs for LAION-2B-en (the red swarms) are higher than the 32 file-level HCRs for the LAION400M (the blue swarms) for all three sub-categories – hatefu