[Paper Review] Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Nitzan Bitton-Guetta, Yonatan Bitton+5 more|arXiv (Cornell University)|Mar 13, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces WHOOPS!, a vision-and-language benchmark featuring 500 synthetic, commonsense-defying images generated via text-to-image models like Midjourney. It evaluates state-of-the-art models on four tasks—including explanation generation—revealing a significant performance gap between humans (95% accuracy) and models like BLIP2-XXL (27%) and GPT-3 (33%), highlighting the need for improved visual commonsense reasoning in AI.

ABSTRACT

Weird, unusual, and uncanny images pique the curiosity of observers because they challenge commonsense. For example, an image released during the 2022 world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo playing chess, which playfully violates our expectation that their competition should occur on the football field. Humans can easily recognize and interpret these unconventional images, but can AI models do the same? We introduce WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. We consider several tasks posed over the dataset. In addition to image captioning, cross-modal matching, and visual question answering, we introduce a difficult explanation generation task, where models must identify and explain why a given image is unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!. We hope our dataset will inspire the development of AI models with stronger visual commonsense reasoning abilities. Data, models and code are available at the project website: whoops-benchmark.github.io

Motivation & Objective

• To develop a benchmark that challenges AI models' visual commonsense reasoning beyond object recognition.
• To create a diverse, synthetic dataset of images that violate physical, temporal, biological, or cultural norms.
• To evaluate state-of-the-art vision-and-language models on novel tasks, especially explanation generation, to expose current limitations.
• To provide a publicly accessible dataset, code, and leaderboard to support systematic evaluation and progress in visual commonsense reasoning.

Proposed method

• Collaborated with designers to generate 500 unusual images using text-to-image models (e.g., Midjourney, DALL-E, Stable Diffusion), modifying plausible scenes to create commonsense violations.
• Annotated each image with detailed explanations of why it is weird, descriptive captions, underspecified captions, and visual question-answer pairs.
• Formulated four benchmark tasks: (1) explanation generation, (2) image captioning, (3) cross-modal matching between detailed and underspecified captions, and (4) visual question answering.
• Introduced a GPT-4-based automatic evaluation metric for explanation generation that aligns with human ratings (81% accuracy).
• Conducted zero-shot and supervised evaluations on models including BLIP2-XXL, OFA, CoCa, and a GPT-3 pipeline, using both synthetic and natural image comparisons.
• Collected contrast sets of (natural, normal, weird) image triplets to isolate the impact of 'weirdness' from synthetic image generation on model performance.

Experimental results

Research questions

• RQ1 Can state-of-the-art vision-and-language models generate plausible explanations for why an image violates commonsense?
• RQ2 How does model performance on explanation generation compare to human performance on commonsense-violating images?
• RQ3 Does the difficulty in WHOOPS! stem from image synthesis or from the inherent weirdness and compositional incongruities in the images?
• RQ4 Which types of commonsense violations (e.g., temporal, biological, physical) are most challenging for current models?
• RQ5 Can automated evaluation metrics for explanation generation reliably reflect human judgments?

Key findings

• Human performance on explanation generation reached 95%, while the best model (fine-tuned BLIP2-XXL) achieved only 27% acceptability, indicating a substantial gap in visual commonsense reasoning.
• The GPT-4-based automatic evaluation metric for explanations achieved 81% accuracy in aligning with human ratings, enabling reliable automated benchmarking.
• Even when provided with a ground-truth image description (oracle input), models only achieved 68% accuracy in explaining the weirdness, proving that the challenge lies in reasoning, not captioning.
• Model performance varied significantly by commonsense category: both BLIP2-XXL and GPT-3 pipeline struggled most with temporal discrepancies and art knowledge violations.
• Image captioning accuracy dropped from 89% on natural and normal images to 49% on weird images, confirming that weirdness—not synthesis—is the primary challenge.
• The pipeline approach using GPT-3 on predicted captions outperformed end-to-end BLIP2-XXL in 46% of commonsense categories, suggesting that large language models can better reason about visual incongruities when guided by intermediate outputs.