[Paper Review] Visual Grounding Strategies for Text-Only Natural Language Processing

Damien Sileo|arXiv (Cornell University)|Mar 25, 2021

Multimodal Machine Learning Applications41 references4 citations

TL;DR

This paper proposes two visual grounding strategies—transferred grounding and associative grounding—to enhance text-only NLP tasks using multimodal pretraining. By integrating visual signals via placeholders or image retrieval, the approach improves performance on language modeling and commonsense reasoning tasks, with associative grounding showing consistent gains across diverse NLP benchmarks.

ABSTRACT

Visual grounding is a promising path toward more robust and accurate Natural Language Processing (NLP) models. Many multimodal extensions of BERT (e.g., VideoBERT, LXMERT, VL-BERT) allow a joint modeling of texts and images that lead to state-of-the-art results on multimodal tasks such as Visual Question Answering. Here, we leverage multimodal modeling for purely textual tasks (language modeling and classification) with the expectation that the multimodal pretraining provides a grounding that can improve text processing accuracy. We propose possible strategies in this respect. A first type of strategy, referred to as {\it transferred grounding} consists in applying multimodal models to text-only tasks using a placeholder to replace image input. The second one, which we call {\it associative grounding}, harnesses image retrieval to match texts with related images during both pretraining and text-only downstream tasks. We draw further distinctions into both strategies and then compare them according to their impact on language modeling and commonsense-related downstream tasks, showing improvement over text-only baselines.

Motivation & Objective

• To investigate whether multimodal pretraining with visual signals can improve performance on purely text-based NLP tasks.
• To address the challenge of adapting visuolinguistic models—originally designed for multimodal tasks—to text-only downstream applications.
• To systematically compare different visual grounding strategies in terms of accuracy, efficiency, and memory usage.
• To evaluate the trade-off between image representation quality and computational cost in multimodal transformers.
• To demonstrate that visual grounding can enhance text understanding beyond what text-only pretraining achieves, especially in commonsense reasoning.

Proposed method

• Proposes 'transferred grounding' where a multimodal model is fine-tuned on text-only tasks using a placeholder for image input, preserving the text encoder's learned representations.
• Introduces 'associative grounding' that retrieves relevant images from a large external collection during both pretraining and inference using an association module.
• Employs a shared multimodal transformer architecture with cross-attention between text and visual features, using region-based image representations from CNNs.
• Applies joint masked language modeling (JMLM) on text and image features, allowing the model to attend to both modalities during pretraining.
• Reduces memory usage by optimizing image representation size and using efficient retrieval mechanisms in the associative strategy.
• Evaluates strategies on standard NLP benchmarks including GLUE, SuperGLUE, and SentEval, focusing on language modeling and commonsense reasoning tasks.

Experimental results

Research questions

• RQ1 Can multimodal pretraining with visual signals improve performance on text-only NLP tasks such as language modeling and text classification?
• RQ2 How do different visual grounding strategies—transferred versus associative—compare in terms of accuracy and efficiency?
• RQ3 What is the impact of image representation quality and retrieval accuracy on downstream NLP performance?
• RQ4 How does the size of image representations affect memory usage and model performance in multimodal transformers?
• RQ5 To what extent can visual grounding enhance commonsense reasoning and plausibility detection beyond text-only baselines?

Key findings

• The associative grounding strategy achieves the best overall performance across language modeling, plausibility estimation, metaphoricity detection, and discourse relation prediction.
• Transferred grounding shows strong results on downstream NLP tasks, particularly because it does not require images at inference time.
• Object-based associative grounding outperforms image-based retrieval in both accuracy and robustness, especially in zero-shot and few-shot settings.
• Reducing image representation size leads to a measurable trade-off between memory efficiency and performance, with optimal balance found at smaller embedding dimensions.
• Joint pretraining with both captioned images and text-only corpora improves generalization and reduces domain bias compared to pretraining only on image-caption pairs.
• Visual grounding leads to consistent improvements on commonsense reasoning tasks, suggesting that visual associations enhance semantic and contextual understanding beyond text-only supervision.