[Paper Review] Robustness Analysis of Video-Language Models Against Visual and Language Perturbations

Madeline C. Schiappa, Shruti Vyas+3 more|arXiv (Cornell University)|Jul 5, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper presents the first comprehensive robustness analysis of video-language models against real-world visual and linguistic perturbations in text-to-video retrieval. Using two large-scale benchmarks—MSRVTT-P and YouCook2-P—featuring 90 visual and 35 text perturbations, the study reveals that models are more vulnerable to video-only perturbations, benefit significantly from pre-training, attend more to objects and scenes than motion, and exhibit gender bias when fine-tuned on target datasets rather than pre-trained ones.

ABSTRACT

Joint visual and language modeling on large-scale datasets has recently shown good progress in multi-modal tasks when compared to single modal learning. However, robustness of these approaches against real-world perturbations has not been studied. In this work, we perform the first extensive robustness study of video-language models against various real-world perturbations. We focus on text-to-video retrieval and propose two large-scale benchmark datasets, MSRVTT-P and YouCook2-P, which utilize 90 different visual and 35 different text perturbations. The study reveals some interesting initial findings from the studied models: 1) models are generally more susceptible when only video is perturbed as opposed to when only text is perturbed, 2) models that are pre-trained are more robust than those trained from scratch, 3) models attend more to scene and objects rather than motion and action. We hope this study will serve as a benchmark and guide future research in robust video-language learning. The benchmark introduced in this study along with the code and datasets is available at https://bit.ly/3CNOly4.

Motivation & Objective

• To investigate the robustness of video-language models against real-world distribution shifts in both visual and textual modalities.
• To identify whether pre-training improves robustness in video-language models compared to training from scratch.
• To analyze model attention mechanisms and determine whether they focus more on static content (objects, scenes) or dynamic content (motion, action).
• To evaluate whether fine-tuning on target datasets introduces or amplifies gender bias in video-language models.
• To establish a benchmark for future research on robust multimodal video-language learning.

Proposed method

• Constructed two large-scale benchmark datasets, MSRVTT-P and YouCook2-P, by applying 90 distinct visual perturbations and 35 text perturbations to the original MSRVTT and YouCook2 datasets.
• Applied diverse perturbations including image blur, camera shake, digital compression, typos, synonym swaps, and positional shifts to simulate real-world distribution shifts.
• Evaluated multiple video-language models—including VideoCLIP, UniVL, and FIT—under various training regimes: from scratch, zero-shot (using pre-trained weights), and fine-tuned on target datasets.
• Measured robustness using relative and absolute robustness scores across different perturbation types and modalities.
• Conducted attention visualization to analyze which visual and linguistic components models attend to under perturbations.
• Performed bias evaluation by analyzing model predictions across gendered terms in text and corresponding video content.

Experimental results

Research questions

• RQ1 How do video-language models perform under real-world visual and linguistic perturbations in text-to-video retrieval?
• RQ2 Does pre-training improve robustness compared to training from scratch?
• RQ3 Do models attend more to static elements (objects, scenes) or dynamic elements (motion, action) under perturbations?
• RQ4 Does fine-tuning on a target dataset introduce or amplify gender bias in video-language models?
• RQ5 How does the robustness of models vary across different types of perturbations (natural, machine-generated, synthetic)?

Key findings

• Models are significantly more susceptible to perturbations when only the video modality is corrupted, indicating higher vulnerability in visual processing compared to textual perturbations.
• Pre-trained models consistently outperform models trained from scratch in both robustness and performance, demonstrating the value of large-scale pre-training.
• Attention mechanisms in video-language models predominantly focus on static visual elements such as objects and scenes, with minimal attention to motion and action cues.
• Models fine-tuned on target datasets exhibit stronger gender biases than those using pre-trained weights, even after fine-tuning, suggesting that target data can amplify existing biases.
• Zero-shot models, particularly UniVL and VideoCLIP, show relatively higher robustness across all perturbation types, especially under synthetic and machine-generated perturbations.
• The average absolute robustness score on MSRVTT-P was 0.82 ± 0.14 for synthetic (DropText/Positional) perturbations, while the best-performing model (UniVL-zs) achieved 0.92 ± 0.05, indicating strong performance under challenging conditions.