[Paper Review] The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Jiayuan Mao, Chuang Gan+3 more|arXiv (Cornell University)|Apr 26, 2019

Multimodal Machine Learning ApplicationsComputer Science52 references312 citations

TL;DR

NS-CL jointly learns visual concepts, word meanings, and semantic parsing from images paired with questions and answers, using a neuro-symbolic framework to perform interpretable, compositional visual reasoning without program annotations. It achieves strong results on CLEVR and generalizes to new attributes, compositions, and domains.

ABSTRACT

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.

Motivation & Objective

• Learn visual concepts, word representations, and semantic parsing from images with QA pairs without explicit annotations for any component.
• Build an object-based scene representation and translate questions into executable programs.
• Use a neuro-symbolic reasoning module to execute programs on latent scene representations for accurate answers.
• Achieve compositional generalization to unseen attributes, scene compositions, and new program domains.
• Demonstrate applicability to VQA and image-text retrieval while enabling interpretable visual concepts.

Proposed method

• Perception module uses Mask R-CNN and ResNet-34 to create object-based scene representations.
• Visual concepts are implemented as neural operators mapping object embeddings to attribute-specific spaces with cosine similarity for classification.
• Semantic parsing translates natural language questions into a hierarchical domain-specific language (DSL) of executable programs.
• Quasi-symbolic program executor deterministically applies program modules to the scene representation; intermediate results are probabilistic masks over objects to maintain differentiability.
• Training optimizes perception and semantic parsing jointly via maximum likelihood over answers; REINFORCE with off-policy search optimizes the parser; curriculum learning guides progression from simple to complex questions.
• Curriculum stages: (1) object-level concepts, (2) relational concepts, (3) complex questions with fixed perception, (4) joint fine-tuning.

Experimental results

Research questions

• RQ1 Can visual concepts, language semantics, and semantic parsing be learned jointly from images and QA pairs without explicit supervision on any component?
• RQ2 Does a neuro-symbolic framework enable interpretable reasoning and robust combinatorial generalization in VQA tasks?
• RQ3 How well does the model generalize to new attributes, compositions, and domains without program annotations?
• RQ4 Can learned concepts transfer to related tasks like image-caption retrieval or other DSL-driven domains?
• RQ5 What is the impact of curriculum learning on convergence and performance across object-level, relational, and complex questions?

Key findings

• NS-CL achieves near-perfect concept quantization for object properties and strong accuracy on visual reasoning with limited data (trained on 5K images with 20 questions per image).
• The model delivers state-of-the-art QA performance on CLEVR without program annotations, and competitive results relative to models with full supervision on program traces.
• NS-CL demonstrates robust combinatorial generalization, adapting to larger scenes and deeper question programs beyond training distribution.
• The learned visual concepts transfer to new tasks such as image-caption retrieval without additional fine-tuning, and to new DSLs with minimal semantic parsing adaptation.
• NS-CL shows strong data efficiency; with only 10% of CLEVR data, it matches baselines trained on full data for certain evaluations.
• Ablations indicate object-based representations with symbolic execution provide interpretable execution traces and effective disentanglement between perception and reasoning.