[Paper Review] Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem

Yudong Han, Liqiang Nie+3 more|arXiv (Cornell University)|Jul 24, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a visual perturbation-aware collaborative learning framework to mitigate the language prior problem in Visual Question Answering (VQA) by enforcing intra-instance invariance and inter-instance discrimination through curated image perturbations. It achieves state-of-the-art performance, improving baseline models by up to 22.91% on VQA-CP v1 and 19.67% on VQA-CP v2, with robustness on balanced benchmarks.

ABSTRACT

Several studies have recently pointed that existing Visual Question Answering (VQA) models heavily suffer from the language prior problem, which refers to capturing superficial statistical correlations between the question type and the answer whereas ignoring the image contents. Numerous efforts have been dedicated to strengthen the image dependency by creating the delicate models or introducing the extra visual annotations. However, these methods cannot sufficiently explore how the visual cues explicitly affect the learned answer representation, which is vital for language reliance alleviation. Moreover, they generally emphasize the class-level discrimination of the learned answer representation, which overlooks the more fine-grained instance-level patterns and demands further optimization. In this paper, we propose a novel collaborative learning scheme from the viewpoint of visual perturbation calibration, which can better investigate the fine-grained visual effects and mitigate the language prior problem by learning the instance-level characteristics. Specifically, we devise a visual controller to construct two sorts of curated images with different perturbation extents, based on which the collaborative learning of intra-instance invariance and inter-instance discrimination is implemented by two well-designed discriminators. Besides, we implement the information bottleneck modulator on latent space for further bias alleviation and representation calibration. We impose our visual perturbation-aware framework to three orthodox baselines and the experimental results on two diagnostic VQA-CP benchmark datasets evidently demonstrate its effectiveness. In addition, we also justify its robustness on the balanced VQA benchmark.

Motivation & Objective

• To address the language prior problem in VQA, where models rely on statistical question-answer correlations rather than visual content.
• To explicitly model how visual cues affect answer representation, moving beyond coarse-grained class-level discrimination.
• To enhance fine-grained instance-level representation learning by enforcing intra-instance invariance and inter-instance discrimination.
• To calibrate learned representations using an information bottleneck modulator for bias reduction and improved generalization.
• To develop a model-agnostic framework that improves existing VQA baselines without requiring task-specific retraining.

Proposed method

• A visual controller dynamically generates two types of perturbed image features—soft and hard perturbations—based on class activation maps.
• Two discriminators are trained to enforce intra-instance invariance (consistency across perturbations) and inter-instance discrimination (distinguishability between different samples).
• An information bottleneck modulator is applied to the latent space to preserve discriminative information while reducing bias.
• The framework integrates with standard VQA models via a collaborative learning scheme that jointly optimizes for invariance and discrimination.
• Negative answer mining is used to improve intra-instance learning, with a hyperparameter controlling the number of top negative candidates.
• Hyperparameters such as λvib and λb are tuned to balance perturbation strength and representation calibration.

Experimental results

Research questions

• RQ1 Can visual perturbation-aware learning effectively reduce language bias in VQA by modeling fine-grained visual effects on answer representations?
• RQ2 Does enforcing intra-instance invariance and inter-instance discrimination improve model robustness and performance on biased benchmarks?
• RQ3 How does the proposed method compare to state-of-the-art approaches on diagnostic VQA-CP datasets?
• RQ4 What is the impact of hyperparameters like N′ and λb on model performance and representation calibration?
• RQ5 Can the framework generalize to standard, balanced VQA benchmarks and improve baseline models?

Key findings

• The proposed method achieves a 1.83% absolute improvement in overall accuracy over SOTA on VQA-CP v1 and 1.86% on VQA-CP v2.
• On VQA-CP v1, the method improves the UpDn baseline by 22.91%, and on VQA-CP v2, it improves UpDn by 19.67%, demonstrating strong generalization.
• The model maintains strong performance on the balanced VQA benchmark, indicating reduced data bias and improved reasoning capacity.
• Performance peaks when N′ (number of negative candidates) is around 20, beyond which performance drops due to inclusion of low-confidence negatives.
• Optimal performance is achieved with λb ≈ 2; values above this drastically reduce performance due to excessive emphasis on invariance.
• Visualization shows the model learns more accurate attention maps focused on relevant visual elements (e.g., frisbee, people), rather than spurious cues like grass or ball.