[Paper Review] T-MARS: Improving Visual Representations by Circumventing Text Feature Learning

Pratyush Maini, Sachin Goyal+3 more|arXiv (Cornell University)|Jul 6, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

T-MARS proposes a novel data filtering method that improves visual representation learning by identifying and removing image-caption pairs where text dominates the visual features, using text masking and CLIP similarity re-scoring. It achieves state-of-the-art performance, outperforming prior methods by 6.5% on ImageNet zero-shot accuracy in the DataComp benchmark and showing linear accuracy gains with data scale.

ABSTRACT

Large web-sourced multimodal datasets have powered a slew of new methods for learning general-purpose visual representations, advancing the state of the art in computer vision and revolutionizing zero- and few-shot recognition. One crucial decision facing practitioners is how, if at all, to curate these ever-larger datasets. For example, the creators of the LAION-5B dataset chose to retain only image-caption pairs whose CLIP similarity score exceeded a designated threshold. In this paper, we propose a new state-of-the-art data filtering approach motivated by our observation that nearly 40% of LAION's images contain text that overlaps significantly with the caption. Intuitively, such data could be wasteful as it incentivizes models to perform optical character recognition rather than learning visual features. However, naively removing all such data could also be wasteful, as it throws away images that contain visual features (in addition to overlapping text). Our simple and scalable approach, T-MARS (Text Masking and Re-Scoring), filters out only those pairs where the text dominates the remaining visual features -- by first masking out the text and then filtering out those with a low CLIP similarity score of the masked image. Experimentally, T-MARS outperforms the top-ranked method on the "medium scale" of DataComp (a data filtering benchmark) by a margin of 6.5% on ImageNet and 4.7% on VTAB. Additionally, our systematic evaluation on various data pool sizes from 2M to 64M shows that the accuracy gains enjoyed by T-MARS linearly increase as data and compute are scaled exponentially. Code is available at https://github.com/locuslab/T-MARS.

Motivation & Objective

• To address the issue of text-dominated images in web-scale vision-language datasets, which encourage optical character recognition (OCR) over visual feature learning.
• To develop a scalable and effective data curation method that preserves visually rich images while filtering out those where text features dominate.
• To improve zero-shot and few-shot vision performance by enhancing data quality through targeted filtering.
• To demonstrate that removing 'bad' examples (text-only images) is more beneficial than adding more data, especially at scale.

Proposed method

• T-MARS first masks all text in images using a pre-trained OCR model to isolate visual features.
• It then computes the CLIP embedding similarity between the masked image and its corresponding caption.
• Pairs with low similarity scores after masking are filtered out, indicating that text was the primary signal for matching.
• The method is scalable and operates efficiently on billion-scale datasets by leveraging pre-computed CLIP features.
• It uses a threshold-based filtering strategy on the masked-image caption similarity, avoiding full retraining.
• The approach is evaluated across multiple data pool sizes (2M to 64M) and benchmarks including DataComp and ImageNet.

Experimental results

Research questions

• RQ1 Does filtering out image-caption pairs where text dominates visual features lead to better visual representation learning?
• RQ2 Can a simple masking-and-re-scoring approach outperform existing data filtering baselines at scale?
• RQ3 How do accuracy gains from data filtering scale with increasing data and compute budgets?
• RQ4 What is the relative utility of different data types—e.g., visual-only, text-only, or visual+text—on downstream vision performance?
• RQ5 Is removing low-quality (text-dominated) examples more beneficial than adding more high-quality data?

Key findings

• T-MARS outperforms the top-ranked method on the 'medium scale' of the DataComp benchmark by 6.5% on ImageNet zero-shot accuracy and 4.7% on VTAB.
• On data pools ranging from 2M to 64M samples, T-MARS shows a linear increase in accuracy gains as data and compute scale exponentially.
• Filtering with T-MARS allows models to outperform training on a CLIP-score-filtered dataset, even with only half the training samples and 50% of the compute.
• Images with only text features have negative utility (U = -0.89), comparable to mislabeled data, and harm downstream performance.
• Images with both visual and text features have nearly the same positive utility (U ≈ +0.23 to +0.27) as those with only visual features, justifying retention.
• Pruning bad examples (text-only images) provides over three times the utility gain of adding new good examples, highlighting the importance of data quality.