[Paper Review] Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

Zhuowan Li, Xingrui Wang+5 more|arXiv (Cornell University)|Dec 1, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces Super-CLEVR, a controllable virtual benchmark to diagnose domain robustness in visual reasoning by isolating four key factors: visual complexity, question redundancy, concept distribution, and concept compositionality. By training and testing VQA models—including a novel probabilistic NSVQA (P-NSVQA) that integrates uncertainty reasoning—on these controlled shifts, the authors show that disentangling perception from reasoning and modeling uncertainty significantly improves robustness, with P-NSVQA outperforming existing methods on three of the four domain shift factors.

ABSTRACT

Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.

Motivation & Objective

• To address the lack of systematic analysis of contributing factors in VQA domain shifts due to intertwined variations in vision, language, and reasoning.
• To develop a controllable virtual benchmark that enables independent study of domain shift factors in visual question answering.
• To evaluate existing VQA models across isolated domain shift factors and diagnose their robustness weaknesses.
• To improve model robustness by integrating uncertainty reasoning into neuro-symbolic VQA frameworks.
• To validate findings on real-world data by transferring insights from synthetic Super-CLEVR to the real GQA dataset.

Proposed method

• Super-CLEVR generates synthetic VQA data using 3D vehicle models from the UDA-Part dataset with controlled variations in visual complexity, question redundancy, concept distribution, and compositionality.
• Questions and answers are automatically generated via templates, enabling precise control over each domain shift factor.
• The benchmark supports four isolated domain shift axes: (1) visual complexity (object count, texture), (2) question redundancy (redundant vs. minimal phrasing), (3) concept distribution (shifted prior over concepts), and (4) concept compositionality (novel combinations not seen in training).
• Four existing models are evaluated: FiLM (non-symbolic), mDETR (large-scale pretrained), NSCL, and NSVQA (neuro-symbolic).
• A new model, probabilistic NSVQA (P-NSVQA), is proposed by injecting uncertainty estimates into the symbolic executor of NSVQA, allowing probabilistic aggregation of reasoning steps.
• Model performance is evaluated under both in-domain and out-of-domain settings across each isolated factor, with ablation and comparison across methods.

Figure 1 : We decompose VQA domain shifts into four contributing factors: visual complexity, question redundancy, concept distribution and concept compositionality. The domain shifts along each factor can be independently studied with the proposed Super-CLEVR dataset.

Experimental results

Research questions

• RQ1 How do different domain shift factors—visual complexity, question redundancy, concept distribution, and compositionality—affect the robustness of VQA models?
• RQ2 To what extent do modular neuro-symbolic methods outperform non-modular models under controlled domain shifts?
• RQ3 Can integrating uncertainty reasoning into symbolic VQA models improve robustness across diverse domain shift factors?
• RQ4 Do findings from synthetic benchmarks like Super-CLEVR generalize to real-world VQA datasets such as GQA?
• RQ5 What specific architectural choices (e.g., disentangled perception/reasoning, uncertainty modeling) lead to improved domain generalization?

Key findings

• P-NSVQA outperforms all baseline models on three of the four domain shift factors: question redundancy, concept distribution, and concept compositionality.
• Neural modular methods like NSVQA show superior robustness to question redundancy compared to non-modular models such as FiLM and mDETR.
• The non-modular mDETR model is more robust to visual complexity shifts than modular models, indicating a trade-off between modularity and robustness to visual complexity.
• On the real GQA dataset, mDETR suffers significantly greater accuracy drops (−14.56%) than neuro-symbolic models (−5.78% for P-NSVQA) as redundant question information is progressively removed, confirming findings from Super-CLEVR.
• The performance drop of mDETR is −14.56% when 100% of redundant information is removed, while P-NSVQA drops only by −5.78%, demonstrating its robustness to question redundancy in real-world settings.
• The study confirms that disentangling perception and reasoning, combined with explicit uncertainty modeling, leads to stronger and more robust VQA models across diverse domain shifts.

Figure 2 : Super-CLEVR contains 21 vehicle models belonging to 5 categories, with controllable attributes.