[Paper Review] Probing Image-Language Transformers for Verb Understanding

Lisa Anne Hendricks, Aida Nematzadeh|arXiv (Cornell University)|Jun 16, 2021

Multimodal Machine Learning Applications26 references4 citations

TL;DR

This paper introduces SVO-Probes, a benchmark of 48,000 image-sentence pairs with controlled negatives for subject, verb, and object, to evaluate verb understanding in image-language transformers. Despite strong downstream performance, models fail most on verb mismatches, overpredicting matches even for semantically unrelated pairs, and perform worse on noisy Conceptual Captions pretraining than on cleaner MSCOCO data, highlighting a critical gap in verb-level multimodal understanding.

ABSTRACT

Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning (e.g., visual question answering and image retrieval). We are interested in shedding light on the quality of their pretrained representations -- in particular, if these models can distinguish different types of verbs or if they rely solely on nouns in a given sentence. To do so, we collect a dataset of image-sentence pairs (in English) consisting of 421 verbs that are either visual or commonly found in the pretraining data (i.e., the Conceptual Captions dataset). We use this dataset to evaluate pretrained image-language transformers and find that they fail more in situations that require verb understanding compared to other parts of speech. We also investigate what category of verbs are particularly challenging.

Motivation & Objective

• To investigate whether image-language transformers can distinguish fine-grained linguistic differences, especially verbs, in multimodal contexts.
• To evaluate if models rely on language priors or truly understand semantic relations between verbs and images.
• To compare the impact of pretraining data quality (noisy vs. clean) on verb understanding in vision-language models.
• To identify which verb categories are most challenging for current models.

Proposed method

• Constructed a new benchmark, SVO-Probes, with 48,100 image-sentence pairs annotated for subject, verb, and object, including controlled negative examples where only one element (subject, verb, or object) differs.
• Designed image-sentence pairs such that each negative example differs in only one aspect (e.g., verb), enabling precise probing of model understanding.
• Evaluated multiple image-language transformer architectures (e.g., MMT) in a zero-shot setting using pretrained weights to classify image-sentence pair matches.
• Compared models fine-tuned on two pretraining datasets: noisy Conceptual Captions (3M pairs) and cleaner, manually-annotated MSCOCO (120K pairs).
• Performed ablation studies by analyzing classification accuracy across subject, verb, and object negatives to isolate performance differences.
• Identified hard and easy verbs by measuring negative pair accuracy across 147 high-frequency verbs with at least 30 negative examples.

Experimental results

Research questions

• RQ1 Can image-language transformers accurately distinguish image-sentence pairs that differ only in the verb, indicating true verb understanding?
• RQ2 Does pretraining on a larger but noisier dataset (Conceptual Captions) degrade verb understanding compared to a smaller, cleaner dataset (MSCOCO)?
• RQ3 Are models more prone to overpredicting matches for semantically related but incorrect image-sentence pairs, especially when the verb is incorrect?
• RQ4 Which verb categories are most difficult for models to disambiguate, and what properties make them challenging?
• RQ5 Do verbs that are considered 'visual' in existing datasets (e.g., imSitu) perform better on this benchmark?

Key findings

• Image-language transformers perform worst on verb-related negatives, with a negative pair accuracy of only 39.5%, significantly lower than for subjects (52.4%) and objects (73.4%).
• Models trained on Conceptual Captions (CC) achieve 64.3% overall accuracy on SVO-Probes, while those trained on MSCOCO achieve 68.0%, despite MSCOCO being smaller and less diverse.
• The MMT model trained on MSCOCO outperforms the CC-pretrained model on negative pair classification, especially for verbs, indicating that cleaner data leads to better sensitivity to semantic mismatches.
• The model overpredicts matches for image-sentence pairs that are only loosely related, suggesting it learns to associate semantically plausible pairs even when the verb does not match.
• Verbs like 'cut', 'argue', and 'break' are among the hardest to classify correctly, while 'tackle', 'lead', and 'surround' are easier, indicating that semantic and perceptual complexity affects model performance.
• Contrary to expectations, verbs labeled as 'visual' in imSitu are harder for the model to classify than non-visual verbs, suggesting that visuality alone does not predict model performance.