[Paper Review] How to Design Sample and Computationally Efficient VQA Models

Karan Samel, Zelin Zhao+4 more|arXiv (Cornell University)|Mar 22, 2021

Multimodal Machine Learning Applications18 references4 citations

TL;DR

This paper proposes DePe, a sample- and computation-efficient VQA model that uses object-level scene graphs for vision and probabilistic programs for language, enabling end-to-end differentiable training with soft logic reasoning. DePe achieves state-of-the-art accuracy while significantly improving sample and computational efficiency over existing methods, particularly under low-data regimes.

ABSTRACT

In multi-modal reasoning tasks, such as visual question answering (VQA), there have been many modeling and training paradigms tested. Previous models propose different methods for the vision and language tasks, but which ones perform the best while being sample and computationally efficient? Based on our experiments, we find that representing the text as probabilistic programs and images as object-level scene graphs best satisfy these desiderata. We extend existing models to leverage these soft programs and scene graphs to train on question answer pairs in an end-to-end manner. Empirical results demonstrate that this differentiable end-to-end program executor is able to maintain state-of-the-art accuracy while being sample and computationally efficient.

Motivation & Objective

• To identify the most sample- and computationally efficient components for vision, language, and reasoning in VQA.
• To investigate the neuro-symbolic trade-off between neural and symbolic modeling in multi-modal reasoning.
• To design a unified VQA model that integrates the most efficient components for improved efficiency without sacrificing accuracy.
• To evaluate the performance of the proposed model under low-data supervision, particularly in terms of sample efficiency.

Proposed method

• Represents images as object-level scene graphs to capture structured visual relationships, improving interpretability and efficiency.
• Represents language questions as probabilistic programs (soft programs) to enable differentiable reasoning over symbolic operations.
• Uses soft logic blocks for reasoning, allowing differentiable execution of programs over visual and linguistic representations.
• Trains the entire model end-to-end using standard backpropagation, avoiding reinforcement learning or discrete sampling.
• Integrates vision, language, and reasoning components via a differentiable program executor that processes scene graphs and soft programs jointly.
• Leverages direct supervision on object and program features (O and FP) at low data scales (0.1% and 1%) to improve sample efficiency.

Experimental results

Research questions

• RQ1 Which combination of vision, language, and reasoning components yields the best balance of sample and computational efficiency in VQA?
• RQ2 How does end-to-end differentiable training compare to reinforcement learning or abduction-based optimization in terms of efficiency and accuracy?
• RQ3 Can probabilistic programs and object-level scene graphs jointly improve model performance while reducing data and compute requirements?
• RQ4 How does the model perform under low-data regimes, particularly when only 0.1% of annotations are used?

Key findings

• DePe achieves 99.0% accuracy on the full VQA dataset (100% data) and 98.0% on 0.1% supervised data, outperforming most baselines under low-data conditions.
• With 10% data, DePe achieves 98.0% accuracy, surpassing models like NS-VQA (92.1%) and TbD (54.2%), demonstrating superior sample efficiency.
• DePe’s computational efficiency is significantly higher than NS-CL, which uses REINFORCE and requires many iterations to converge.
• Stack-NMN and Prob-NMN show strong performance under low data but are outperformed by DePe when direct supervision is applied to object and program features.
• The model maintains state-of-the-art accuracy while reducing reliance on expensive discrete sampling or abduction, enabling faster convergence.
• DePe’s end-to-end differentiable training framework enables stable optimization without the high variance seen in discrete-reasoning models like NS-VQA.