[Paper Review] Programmatically Grounded, Compositionally Generalizable Robotic Manipulation

Renhao Wang, Jiayuan Mao+4 more|arXiv (Cornell University)|Apr 26, 2023

Robot Manipulation and Learning4 citations

TL;DR

This paper proposes programport, a program-based modular framework that disentangles vision-language grounding from task-specific policy learning in robotic manipulation by parsing natural language instructions into executable programs using Combinatory Categorial Grammar (CCG). By grounding language concepts and actions through dedicated neural modules, the model achieves superior zero-shot and compositional generalization across unseen object-task combinations, outperforming standard fine-tuning approaches with less data.

ABSTRACT

Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills. Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities. However, we show that the conventional pretraining-finetuning pipeline for integrating such representations entangles the learning of domain-specific action information and domain-general visual information, leading to less data-efficient training and poor generalization to unseen objects and tasks. To this end, we propose ProgramPort, a modular approach to better leverage pretrained VL models by exploiting the syntactic and semantic structures of language instructions. Our framework uses a semantic parser to recover an executable program, composed of functional modules grounded on vision and action across different modalities. Each functional module is realized as a combination of deterministic computation and learnable neural networks. Program execution produces parameters to general manipulation primitives for a robotic end-effector. The entire modular network can be trained with end-to-end imitation learning objectives. Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors. Project webpage at: \url{https://progport.github.io}.

Motivation & Objective

• To address the poor zero-shot and compositional generalization of vision-language (VL) models in robotic manipulation due to entangled learning of perception and control.
• To disentangle domain-general visual grounding from domain-specific policy learning using structured program representations.
• To improve data efficiency and generalization by leveraging pretrained vision-language models through a modular, program-execution pipeline.
• To enable robust reasoning over novel combinations of unseen objects and manipulation behaviors without fine-tuning.
• To develop a framework that faithfully grounds language to vision and action using a domain-specific language (DSL) and executable programs.

Proposed method

• Parse natural language instructions into executable manipulation programs using Combinatory Categorial Grammar (CCG), enabling structured semantic parsing.
• Define a domain-specific language (DSL) with functional modules for visual grounding (e.g., locate objects by category) and action policies (e.g., generate pick/place poses).
• Implement each functional module as a hybrid of deterministic computation and learnable neural networks, allowing end-to-end training with imitation learning objectives.
• Use a pretrained vision-language model (e.g., CLIP) to ground language descriptors to visual inputs in a disentangled manner, independent of task-specific policies.
• Execute the program at inference time to generate control parameters for robotic primitives such as pick and place, based on vision and language inputs.
• Train the entire modular network end-to-end via imitation learning on expert demonstrations, enabling data-efficient policy learning.

Figure 1: Zero-Shot and Compositional Generalization: Our framework, programport , is capable of generalizing to combinations of unseen objects and manipulation behaviors at test time.

Experimental results

Research questions

• RQ1 Can a program-based modular framework improve zero-shot generalization in robotic manipulation compared to standard fine-tuning of vision-language models?
• RQ2 Does disentangling visual grounding from policy learning lead to better compositional generalization across novel object-task combinations?
• RQ3 Can structured program representations derived from natural language improve grounding fidelity and data efficiency in vision-language manipulation?
• RQ4 To what extent can a model generalize to unseen object categories and novel compositions of known concepts without fine-tuning?
• RQ5 How does program-based grounding compare to end-to-end fine-tuning in terms of robustness to out-of-distribution language and visual inputs?

Key findings

• The model achieves significant zero-shot generalization to unseen combinations of objects and manipulation behaviors, such as pushing or placing unfamiliar shapes, without any fine-tuning.
• Programport demonstrates improved data efficiency, learning performant imitation policies with fewer demonstrations compared to standard fine-tuning baselines.
• The disentangled architecture enables faithful grounding of language to vision, avoiding shortcuts that lead to overfitting in conventional fine-tuning approaches.
• The model generalizes to novel compositions of known concepts (e.g., 'push the red cube to the blue box') even when such combinations were not seen during training.
• The use of CCG parsing and a DSL enables structured reasoning over language, resulting in more reliable and interpretable policy execution.
• Empirical results across 10 diverse tabletop manipulation tasks confirm that programport outperforms strong baselines in both zero-shot and compositional generalization settings.

Figure 2: (a) Existing manipulation models entangle visual grounding from pretrained VL models with task-specific policy learning. (b) Our disentangled approach leverages program-based representation of language semantics to yield faithful VL-grounding and improved genearlization.