[Paper Review] Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Kexin Yi, Jia-Jun Wu+4 more|arXiv (Cornell University)|Oct 4, 2018

Multimodal Machine Learning ApplicationsComputer Science48 references234 citations

TL;DR

NS-VQA combines neural scene parsing with a symbolic program executor to reason over a structured scene representation, achieving near-perfect CLEVR accuracy and data-efficient learning with interpretable reasoning.

ABSTRACT

We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

Motivation & Objective

• Motivate disentangling visual perception and language understanding from reasoning in VQA.
• Propose a neural-symbolic architecture that parses scenes and questions into a symbolic program for execution.
• Demonstrate data efficiency, memory efficiency, and interpretability of symbolic execution on CLEVR and related datasets.

Proposed method

• Scene parser (de-renderer) uses Mask R-CNN to generate object proposals and predict attributes, then uses a ResNet-34 backbone on cropped segments for spatial attributes.
• Question parser (program generator) is an attention-based seq2seq model (bidirectional LSTM encoder, LSTM decoder with attention) that maps questions to hierarchical programs.
• Program executor applies a deterministic set of functional modules to the structural scene representation according to the generated program to produce answers.
• Training involves supervised pretraining of the question parser on a small set of (question, program) pairs, followed by REINFORCE fine-tuning on (question, answer) pairs.
• The executable program is fully symbolic and transparent, with modules arranged in a sequence starting from a scene token; errors trigger random sampling of outputs.
• Memory efficiency is achieved by using compact structural representations (less than 100 bytes per image) compared to attention-based baselines.

Experimental results

Research questions

• RQ1 Can a neural-symbolic VQA system disentangle perception, language understanding, and reasoning while preserving accuracy?
• RQ2 How data-efficient is a largely symbolic reasoning pipeline when learning from limited program annotations?
• RQ3 Does symbolic execution improve interpretability and generalization to unseen attribute combinations and human-generated questions?
• RQ4 Can the approach generalize to new visual domains (e.g., Minecraft) and maintain reasoning capabilities?

Key findings

• NS-VQA achieves near-perfect accuracy on CLEVR (up to 99.8% with 270 program annotations) surpassing prior methods.
• The method requires significantly less memory for offline QA (structural representations <100 bytes per image vs. ~20KB for attention-based methods).
• NS-VQA recovers underlying programs with high accuracy, especially as the number of pretraining programs increases (e.g., 88% program accuracy with 500 annotations; near-perfect with 9K).
• The model generalizes to unseen attribute combinations (CLEVR-CoGenT) and human-generated questions (CLEVR-Humans) with limited supervision.
• NS-VQA extends to different scene contexts (Minecraft) with comparable reasoning ability, though occlusion remains a challenge.