[Paper Review] Neural Task Programming: Learning to Generalize Across Hierarchical Tasks

Danfei Xu, Suraj Nair+5 more|arXiv (Cornell University)|Oct 4, 2017

Robot Manipulation and Learning1 references19 citations

TL;DR

Neural Task Programming (NTP) is a meta-learning framework that enables robots to generalize across hierarchical manipulation tasks by interpreting task demonstrations and recursively decomposing them into reusable neural subroutines. It achieves strong few-shot generalization to unseen tasks with varying lengths, topologies, and semantics in both simulation and real-world robot environments, particularly excelling when trained end-to-end with visual inputs.

ABSTRACT

In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction. NTP takes as input a task specification (e.g., video demonstration of a task) and recursively decomposes it into finer sub-task specifications. These specifications are fed to a hierarchical neural program, where bottom-level programs are callable subroutines that interact with the environment. We validate our method in three robot manipulation tasks. NTP achieves strong generalization across sequential tasks that exhibit hierarchal and compositional structures. The experimental results show that NTP learns to generalize well to- wards unseen tasks with increasing lengths, variable topologies, and changing objectives.

Motivation & Objective

• To address the challenge of generalizing robot policies across complex, hierarchical manipulation tasks with varying objectives and initial conditions.
• To enable one-shot learning from demonstrations by discovering reusable, modular neural programs that generalize across task variations.
• To support long-horizon decision making through hierarchical decomposition of tasks into executable subroutines.
• To enable end-to-end training from visual inputs (e.g., videos) without explicit state supervision.
• To improve robustness and transferability in real-world robotic deployment across diverse task structures.

Proposed method

• NTP interprets a task demonstration (e.g., video or trajectory) as a hierarchical program specification and recursively decomposes it into sub-tasks.
• A hierarchical neural program is instantiated, where bottom-level programs correspond to executable robot API actions (e.g., pick_and_place).
• Each program call takes the current environment observation and a sub-specification, predicting the next sub-program and sub-task input, with an end-of-program signal to terminate execution.
• The model uses a recurrent architecture (e.g., GRU) to maintain internal state and support long-term dependencies across sub-tasks.
• For visual input, NTPVID (E2E) jointly learns a policy and task-relevant visual features via a 7-layer CNN, while NTPVID (Detector) uses a pre-trained object detector as a state preprocessor.
• The framework supports meta-learning across tasks by sharing parameters across subroutines, enabling transfer to unseen task instances with minimal data.

Experimental results

Research questions

• RQ1 Can a neural program induction framework generalize to unseen hierarchical tasks with varying lengths and topologies?
• RQ2 Can NTP achieve effective few-shot generalization from a single demonstration across diverse task objectives?
• RQ3 How well does NTP perform when trained end-to-end from visual inputs (e.g., videos) without ground-truth state supervision?
• RQ4 Does NTP maintain robustness under environmental perturbations such as intermittent failures or dynamic changes?
• RQ5 Can NTP be successfully transferred to real-world robotic systems with physical interaction constraints?

Key findings

• NTP achieved 90% success rate on 20 unseen block-stacking tasks on the real Sawyer robot, with only 5% failures due to algorithmic errors and 5% due to physical manipulation issues.
• In the object sorting task, NTP achieved 80% success on 10 unseen real-world evaluations, with no algorithmic failures but 20% manipulation failures.
• NTPVID (E2E), the end-to-end visual model, outperformed NTPVID (Detector) with a pre-trained detector, despite using only partial visual state information.
• The end-to-end visual model (NTPVID (E2E)) achieved higher success rates than the detector-based pipeline, even though it was trained on fewer tasks (400–1000 vs. 1000) and required significantly more compute (10 days on 8 GPUs).
• In adversarial environments with intermittent failures, NTP with GRU dropped from 88.4% to 42.2% success, indicating sensitivity to dynamic disturbances, while standard NTP remained more robust.
• NTP generalized effectively to table-cleanup tasks with varying numbers of bowls and forks, maintaining success rates between 55% and 100% in simulation, with performance degradation attributed to collision-checking errors rather than policy mistakes.