[논문 리뷰] Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Kexin Yi, Jia-Jun Wu외 4명|arXiv (Cornell University)|2018. 10. 04.

Multimodal Machine Learning Applications참고 문헌 48인용 수 234

한 줄 요약

NS-VQA는 신경망 장면 해석과 기호적 프로그램 실행기를 결합하여 구조화된 장면 표현을 추론하고, CLEVR에서 거의 완벽에 가까운 정확도와 해석 가능한 추론으로 데이터 효율적으로 학습한다.

ABSTRACT

We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

연구 동기 및 목표

• VQA에서 시각 지각과 언어 이해를 추론으로부터 분리하는 것을 동기로 삼는다.
• 장면과 질문을 실행을 위한 기호적 프로그램으로 파싱하는 신경-기호 아키텍처를 제안한다.
• 데이터 효율성, 메모리 효율성, 그리고 CLEVR 및 관련 데이터셋에서의 기호적 실행의 해석 가능성을 보여준다.

제안 방법

• Scene parser (de-renderer) uses Mask R-CNN to generate object proposals and predict attributes, then uses a ResNet-34 backbone on cropped segments for spatial attributes.
• Question parser (program generator) is an attention-based seq2seq model (bidirectional LSTM encoder, LSTM decoder with attention) that maps questions to hierarchical programs.
• Program executor applies a deterministic set of functional modules to the structural scene representation according to the generated program to produce answers.
• Training involves supervised pretraining of the question parser on a small set of (question, program) pairs, followed by REINFORCE fine-tuning on (question, answer) pairs.
• The executable program is fully symbolic and transparent, with modules arranged in a sequence starting from a scene token; errors trigger random sampling of outputs.
• Memory efficiency is achieved by using compact structural representations (less than 100 bytes per image) compared to attention-based baselines.

실험 결과

연구 질문

• RQ1 Can a neural-symbolic VQA system disentangle perception, language understanding, and reasoning while preserving accuracy?
• RQ2 How data-efficient is a largely symbolic reasoning pipeline when learning from limited program annotations?
• RQ3 Does symbolic execution improve interpretability and generalization to unseen attribute combinations and human-generated questions?
• RQ4 Can the approach generalize to new visual domains (e.g., Minecraft) and maintain reasoning capabilities?

주요 결과

Methods	Count	Exist	Compare	Compare	Query	Overall
Humans (Johnson et al., 2017b)	86.7	96.6	86.4	96.0	95.0	92.6
CNN+LSTM+SAN (Johnson et al., 2017b)	59.7	77.9	75.1	70.8	80.9	73.2
N2NMN * (Hu et al., 2017)	68.5	85.7	84.9	88.7	90.0	83.7
Dependency Tree (Cao et al., 2018)	81.4	94.2	81.6	97.1	90.5	89.3
CNN+LSTM+RN (Santoro et al., 2017)	90.1	97.8	93.6	97.1	97.9	95.5
IEP * (Johnson et al., 2017b)	92.7	97.1	98.7	98.9	98.1	96.9
CNN+GRU+FiLM (Perez et al., 2018)	94.5	99.2	93.8	99.0	99.2	97.6
DDRprog * (Suarez et al., 2018)	96.5	98.8	98.4	99.0	99.1	98.3
MAC (Hudson and Manning, 2018)	97.1	99.5	99.1	99.5	99.5	98.9
TbD+reg+hres * (Mascharka et al., 2018)	97.6	99.2	99.4	99.6	99.5	99.1
NS-VQA (ours, 90 programs)	64.5	87.4	53.7	77.4	79.7	74.4
NS-VQA (ours, 180 programs)	85.0	92.9	83.4	90.6	92.2	89.5
NS-VQA (ours, 270 programs)	99.7	99.9	99.9	99.8	99.8	99.8

• NS-VQA achieves near-perfect accuracy on CLEVR (up to 99.8% with 270 program annotations) surpassing prior methods.
• The method requires significantly less memory for offline QA (structural representations <100 bytes per image vs. ~20KB for attention-based methods).
• NS-VQA recovers underlying programs with high accuracy, especially as the number of pretraining programs increases (e.g., 88% program accuracy with 500 annotations; near-perfect with 9K).
• The model generalizes to unseen attribute combinations (CLEVR-CoGenT) and human-generated questions (CLEVR-Humans) with limited supervision.
• NS-VQA extends to different scene contexts (Minecraft) with comparable reasoning ability, though occlusion remains a challenge.