[Paper Review] VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models

Wangchunshu Zhou, Yan Zeng+2 more|arXiv (Cornell University)|May 30, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces VLUE, a multi-task benchmark evaluating vision-language models on generalization and efficiency-performance trade-offs. It features out-of-distribution test sets from culturally diverse images, revealing significant generalization gaps and demonstrating that efficiency-aware evaluation reveals superior model trade-offs beyond accuracy alone.

ABSTRACT

Recent advances in vision-language pre-training (VLP) have demonstrated impressive performance in a range of vision-language (VL) tasks. However, there exist several challenges for measuring the community's progress in building general multi-modal intelligence. First, most of the downstream VL datasets are annotated using raw images that are already seen during pre-training, which may result in an overestimation of current VLP models' generalization ability. Second, recent VLP work mainly focuses on absolute performance but overlooks the efficiency-performance trade-off, which is also an important indicator for measuring progress. To this end, we introduce the Vision-Language Understanding Evaluation (VLUE) benchmark, a multi-task multi-dimension benchmark for evaluating the generalization capabilities and the efficiency-performance trade-off (``Pareto SOTA'') of VLP models. We demonstrate that there is a sizable generalization gap for all VLP models when testing on out-of-distribution test sets annotated on images from a more diverse distribution that spreads across cultures. Moreover, we find that measuring the efficiency-performance trade-off of VLP models leads to complementary insights for several design choices of VLP. We release the VLUE benchmark to promote research on building vision-language models that generalize well to more diverse images and concepts unseen during pre-training, and are practical in terms of efficiency-performance trade-off.

Motivation & Objective

• To address the overestimation of vision-language model generalization due to in-domain evaluation on COCO/VG datasets.
• To measure the efficiency-performance trade-off in vision-language pre-training, promoting practical model deployment.
• To establish a standardized, multi-dimensional benchmark for tracking progress beyond absolute performance.
• To provide a publicly available benchmark with OOD test sets to encourage research on robust and efficient vision-language models.

Proposed method

• Annotating a new out-of-distribution (OOD) test set using images from the MaRVL dataset, collected across cultures by native speakers.
• Maintaining identical annotation protocols between in-domain and OOD sets to ensure fair comparison of label distributions.
• Benchmarking 7 representative vision-language models across four tasks: image-text retrieval, VQA, visual reasoning, and visual grounding.
• Measuring inference time on a fixed hardware setup (1x V100 GPU, batch size 1) to evaluate efficiency-performance trade-offs.
• Using the Pareto front to compare models across performance and inference time, identifying models that outperform others on both dimensions.
• Releasing the benchmark, data, and code publicly to support reproducibility and community-wide evaluation.

Experimental results

Research questions

• RQ1 How do vision-language models perform when evaluated on out-of-distribution images from culturally diverse sources?
• RQ2 To what extent does in-domain performance correlate with out-of-distribution generalization performance?
• RQ3 How do different vision feature extraction methods (object detection vs. vision transformers) affect the efficiency-performance trade-off?
• RQ4 Can a multi-dimensional benchmark like VLUE reveal more nuanced insights than absolute performance alone?

Key findings

• A significant generalization gap exists across all evaluated models when tested on OOD test sets, indicating poor robustness to distribution shift.
• The Pearson correlation between in-domain and OOD performance is 0.75, suggesting a moderate but not perfect relationship, highlighting the need for OOD evaluation.
• Models using vision transformers as image encoders achieve better efficiency-performance trade-offs than those relying on object detection features.
• VL-T5 shows a 2.4× longer average inference time than ALBEF despite only marginally lower performance, demonstrating the importance of efficiency-aware evaluation.
• Image captioning models like OFA and VL-T5 suffer substantial performance drops (e.g., CIDER drops from 140.7 to 71.2) on OOD sets, suggesting reliance on COCO-specific features.
• The benchmark reveals that Pareto-optimal models exist that outperform SOTA models in both performance and efficiency, challenging the dominance of accuracy-only metrics.