Skip to main content
QUICK REVIEW

[Paper Review] Data Filtering Networks

Alex Chengyu Fang, Albin Madappally Jose|arXiv (Cornell University)|Sep 29, 2023
Multimodal Machine Learning ApplicationsComputer Science15 citations
TL;DR

The paper learns data filtering networks (DFNs) to curate large image–text datasets; DFN-2B/DFN-5B induce high-quality data that enable state-of-the-art CLIP models at various compute budgets, including 84.4% ImageNet zero-shot accuracy with ViT-H/14.

ABSTRACT

Large training sets have become a cornerstone of machine learning and are the foundation for recent advances in language modeling and multimodal learning. While data curation for pre-training is often still ad-hoc, one common paradigm is to first collect a massive pool of data from the Web and then filter this candidate pool down to an actual training set via various heuristics. In this work, we study the problem of learning a data filtering network (DFN) for this second step of filtering a large uncurated dataset. Our key finding is that the quality of a network for filtering is distinct from its performance on downstream tasks: for instance, a model that performs well on ImageNet can yield worse training sets than a model with low ImageNet accuracy that is trained on a small amount of high-quality data. Based on our insights, we construct new data filtering networks that induce state-of-the-art image-text datasets. Specifically, our best performing dataset DFN-5B enables us to train state-of-the-art CLIP models for their compute budgets: among other improvements on a variety of tasks, a ViT-H trained on our dataset achieves 84.4% zero-shot transfer accuracy on ImageNet, out-performing models trained on other datasets such as LAION-2B, DataComp-1B, or OpenAI's WIT. In order to facilitate further research in dataset design, we also release a new 2 billion example dataset DFN-2B and show that high performance data filtering networks can be trained from scratch using only publicly available data.

Motivation & Objective

  • Motivate data-centric improvements by focusing on the filtering stage of large-scale dataset construction.
  • Characterize features of DFNs that yield high-quality training datasets.
  • Demonstrate that DFNs can be trained from scratch on publicly available data and induce superior datasets for CLIP training.
  • Show that datasets induced by DFNs achieve better trade-offs between accuracy and compute.
  • Provide publicly reproducible DFN pipelines and datasets to advance dataset design research.

Proposed method

  • Define data filtering networks (DFNs) as pointwise filters applied to elements of a large candidate pool.
  • Evaluate DFNs using the DataComp benchmark across multiple pool sizes (medium, large, xlarge) with corresponding CLIP model hyperparameters.
  • Use CLIP-based filtering as a primary DFN backbone, and compare with alternative filters (binary classifiers, M3AE) to assess robustness of the filtering approach.
  • Train DFNs from high-quality data and apply them to filter large uncurated pools to induce training datasets.
  • Fine-tune and ensemble DFNs with standard ML techniques (augmentation, initialization, training steps) to enhance dataset quality.
  • Assess induced models on 38 zero-shot and retrieval tasks within DataComp, reporting ImageNet (IN), IN shifts, VTAB, retrieval, and average metrics.
Figure 1 : Compute scaling behavior of training CLIP models on various datasets. DFN-2B, the subset of CommonPool (DataComp-12.8B) chosen by our best performing data filtering networks, out-performs all other datasets including OpenAI’s WIT and the previous state-of-the-art CLIP training dataset Dat
Figure 1 : Compute scaling behavior of training CLIP models on various datasets. DFN-2B, the subset of CommonPool (DataComp-12.8B) chosen by our best performing data filtering networks, out-performs all other datasets including OpenAI’s WIT and the previous state-of-the-art CLIP training dataset Dat

Experimental results

Research questions

  • RQ1What properties of data filtering networks lead to high-quality induced datasets for CLIP-style models?
  • RQ2Is higher ImageNet performance of a filtering model predictive of better data filtering quality and downstream dataset utility?
  • RQ3Can DFNs be trained from scratch on publicly available data to induce state-of-the-art datasets without relying on proprietary filters?
  • RQ4How do DFN-induced datasets perform across ImageNet, distribution shifts, VTAB, and retrieval tasks compared to existing datasets?
  • RQ5What regimen (augmentation, initialization, training steps) most effectively improves DFN-induced dataset quality?

Key findings

  • A small contrastive image–text model trained on high-quality data suffices to construct state-of-the-art datasets.
  • Filtering strength is not correlated with image task performance; higher ImageNet accuracy of a filtering model does not guarantee better filtering results.
  • Using HQITP-like high-quality data for DFN training yields superior induced datasets across IN, IN shifts, VTAB, and retrieval.
  • DFN-induced datasets (DFN-2B/DFN-5B) achieve state-of-the-art results on DataComp scales, surpassing LAION-2B, DC-1B, OpenAI WIT, and similar baselines.
  • ViT-L/14 trained on DFN-2B attains 81.4% zero-shot ImageNet accuracy; ViT-H/14 trained on DFN-5B attains 84.4% zero-shot ImageNet accuracy; both outperform competing datasets at corresponding compute budgets.
  • Datasets induced by DFNs improve vision tasks and VQA (BLIP2) robustness and performance compared to baselines.
Figure 2 : A high level overview of our pipeline for constructing datasets using DFNs
Figure 2 : A high level overview of our pipeline for constructing datasets using DFNs

Read this paper

Open the full text in the Nubint reader and ask about any passage as you read.

Read this paper on Nubint

This review was generated by AI from the original paper. Please check the paper before citing it.