[Paper Review] Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering

Ramakrishna Vedantam, Karan Desai+4 more|arXiv (Cornell University)|Feb 21, 2019

Multimodal Machine Learning Applications32 citations

TL;DR

The paper introduces Prob-NMN, a probabilistic neural-symbolic model for VQA where latent symbolic programs generate answers via instantiated neural modules. It shows data-efficient legibility and enables probing coherence and sensitivity of reasoning, achieving improvements on CLEVR and SHAPES in low-supervision regimes.

ABSTRACT

We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring lesser number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the programs that could lead to a specified answer given an image. Our results on the CLEVR and SHAPES datasets verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.

Motivation & Objective

• Introduce a probabilistic neural-symbolic framework for VQA where symbolic programs are latent variables.
• Achieve legible, human-understandable reasoning with limited program supervision.
• Enable probing of the model's reasoning via coherence and sensitivity tests.
• Demonstrate improved performance in low-data regimes on CLEVR and SHAPES datasets.

Proposed method

• Define p(x, z, a | i) = p(z) p(x | z) p(a | i; θ_z) with z as a latent program and neural modules θ_z instantiated per program symbol.
• Use an inference network q_φ(z | x) to map questions to latent programs.
• Train via a three-stage process: Question Coding, Module Training, and Joint Training.
• Derive variational lower bounds for semi-supervised and supervised settings and employ a staged optimization strategy.
• Parameterize p(z), p(x|z), qφ(z|x) with LSTMs; instantiate neural modules θ_z per program symbol; pretrain the program prior p(z).
• Employ scaling factors α, β, γ and REINFORCE for non-differentiable components to optimize the full objective.

Experimental results

Research questions

• RQ1 Can Prob-NMN learn meaningful program representations with limited question-program supervision?
• RQ2 Does incorporating probabilistic latent programs improve data efficiency and legibility of the reasoning process?
• RQ3 Can the model’s latent reasoning be interrogated for coherence (consistency across programs leading to the same answer) and sensitivity (changes in answer reflected in the reasoning)?
• RQ4 Do the proposed methods improve VQA accuracy and program prediction on CLEVR and SHAPES under low supervision?

Key findings

• Prob-NMN consistently improves performance in data-sparse regimes over non-probabilistic baselines.
• Question coding with Prob-NMN achieves substantially higher program prediction accuracy than baselines under limited supervision.
• Joint and module training lead to better VQA accuracy and higher quality (more legible) programs than prior neural-symbolic adaptations.
• On SHAPES, Prob-NMN attains 30–35 point higher test VQA accuracy than NMN when program supervision exceeds 5%.
• Prob-NMN enables probing for coherence and sensitivity by sampling latent programs p(z | i, a) to test reasoning consistency across similar/different answers.