[Paper Review] Learning Neuro-symbolic Programs for Language Guided Robot Manipulation

Namasivayam Kalithasan, Himanshu Singh+6 more|arXiv (Cornell University)|Nov 12, 2022

Multimodal Machine Learning Applications33 references4 citations

TL;DR

This paper proposes a neuro-symbolic framework that learns grounded manipulation programs from natural language instructions and scene images, using a hierarchical parser and action simulator to enable end-to-end training without sub-goal supervision. The model achieves state-of-the-art generalization on unseen scenes and longer-horizon tasks by learning disentangled, object-centric representations through symbolic reasoning over visual and linguistic inputs.

ABSTRACT

Given a natural language instruction and an input scene, our goal is to train a model to output a manipulation program that can be executed by the robot. Prior approaches for this task possess one of the following limitations: (i) rely on hand-coded symbols for concepts limiting generalization beyond those seen during training [1] (ii) infer action sequences from instructions but require dense sub-goal supervision [2] or (iii) lack semantics required for deeper object-centric reasoning inherent in interpreting complex instructions [3]. In contrast, our approach can handle linguistic as well as perceptual variations, end-to-end trainable and requires no intermediate supervision. The proposed model uses symbolic reasoning constructs that operate on a latent neural object-centric representation, allowing for deeper reasoning over the input scene. Central to our approach is a modular structure consisting of a hierarchical instruction parser and an action simulator to learn disentangled action representations. Our experiments on a simulated environment with a 7-DOF manipulator, consisting of instructions with varying number of steps and scenes with different number of objects, demonstrate that our model is robust to such variations and significantly outperforms baselines, particularly in the generalization settings. The code, dataset and experiment videos are available at https://nsrmp.github.io

Motivation & Objective

• Address the challenge of grounding complex, multi-step natural language instructions into executable robot manipulation programs without intermediate supervision.
• Overcome limitations of prior methods that rely on hand-coded symbols, require dense sub-goal annotations, or lack deep semantic reasoning for spatial and relational concepts.
• Enable robust generalization to novel scenes with more objects and longer instruction sequences, particularly in combinatorial zero-shot settings.
• Learn dense, disentangled representations of actions and spatial concepts directly from initial and final scene states using only distant supervision.
• Develop a modular, end-to-end trainable architecture that integrates symbolic reasoning with neural perception and action prediction.

Proposed method

• Use a visual extractor with a pre-trained object detector and feature encoder to generate dense, object-centric representations from input scene images.
• Implement a hierarchical language reasoner that auto-regressively constructs symbolic programs from instructions using a domain-specific language (DSL) for spatial and action concepts.
• Integrate a visual reasoner that applies symbolic and spatial reasoning to identify which objects are affected by actions in the predicted program.
• Employ an action simulator that predicts the final 3D location of moved objects based on the symbolic program and visual state, with loss backpropagated through the action and visual modules.
• Train the language reasoner via REINFORCE with policy gradient optimization to sample symbolic programs, enabling differentiable policy learning.
• Train the entire model end-to-end using bounding box loss on predicted object locations, enabling supervision from only initial and final world states.

Experimental results

Research questions

• RQ1 Can a neuro-symbolic model learn to ground complex, multi-step language instructions into executable manipulation programs without sub-goal supervision?
• RQ2 How well does the model generalize to scenes with more objects than seen during training, particularly in combinatorial zero-shot settings?
• RQ3 To what extent can the model handle longer-horizon instructions (e.g., 7-step tasks) after being trained on shorter ones?
• RQ4 Can disentangled, dense action representations be learned end-to-end from only initial and final scene states?
• RQ5 Does symbolic reasoning over object-centric representations improve robustness and accuracy in language-guided manipulation?

Key findings

• The proposed model significantly outperforms neural baselines such as CLIPort and NMN+ in both placement and identification accuracy, especially in zero-shot generalization to scenes with up to 10 objects.
• On scenes with more than 5 objects, the model maintains high accuracy (over 90% in some settings), demonstrating strong combinatorial generalization due to object-centric representation and modular reasoning.
• The model generalizes effectively to longer-horizon tasks—achieving high accuracy on 7-step instructions after being trained only on 2-step plans—highlighting its scalability and structured reasoning capability.
• The action simulator achieves a structural similarity index (SSIM) of 0.935 for object reconstruction, indicating high fidelity in predicting object states after actions.
• The model shows robust performance on complex, relational instructions (e.g., 'put the block behind the green dice to the right of the red cube') where baselines fail due to limited semantic reasoning.
• Even when trained on 5x more data, CLIPort-5x underperforms the proposed model, confirming the advantage of the neuro-symbolic, disentangled architecture over purely neural approaches.