[Paper Review] A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes

Max Bajracharya, James Borders+11 more|arXiv (Cornell University)|Sep 30, 2019

Robot Manipulation and Learning43 references4 citations

TL;DR

This paper presents a mobile manipulation system that enables a robot to learn complex, human-level tasks in real homes through a single virtual reality (VR) demonstration. By combining parameterized hybrid control, learned dense visual embeddings for scene understanding, and a task graph architecture, the system achieves 85% success rate across 60 end-to-end task executions involving an average of 45 behaviors per task, demonstrating robustness to environmental variations without prior maps or object models.

ABSTRACT

We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of tasks, under different environmental conditions, in multiple real homes. Our approach achieves 85% overall success rate on three tasks that consist of an average of 45 behaviors each.

Motivation & Objective

• To enable a general-purpose mobile manipulator to perform complex, human-level tasks in unstructured home environments without prior object models or maps.
• To develop a system that learns new tasks from a single human demonstration in virtual reality, minimizing engineering effort for new tasks.
• To achieve robust execution in real homes under natural environmental variations such as lighting changes, clutter, and object repositioning.
• To decouple perception from behavior through learned visual embeddings, enabling modular, diagnosable task execution.
• To demonstrate end-to-end task success in real homes with minimal human intervention and high fault tolerance.

Proposed method

• The system uses a highly capable mobile manipulator with high end-effector manipulability and wide visual field-of-view to facilitate VR-based teaching.
• Tasks are taught via virtual reality, where users demonstrate behaviors using parameterized, hybrid position/force control to ensure robustness and minimal parameter tuning.
• Dense, learned pixelwise visual embeddings are extracted from keyframes during teaching to enable robust scene matching at execution time, even under viewpoint changes and lighting variations.
• Behaviors are linked to visual embeddings and organized into a dynamic task graph with defined entry conditions and success-based exit criteria, enabling reuse and chaining of behaviors.
• The system avoids explicit global mapping by relying on visual keyframe matching and reactive path following for navigation and manipulation.
• Fault detection and recovery are integrated into the task graph, allowing the robot to detect and recover from 99.6% of behavior failures, reducing catastrophic failures.

Experimental results

Research questions

• RQ1 Can a mobile manipulation robot learn complex, multi-step tasks in real homes from a single VR demonstration without prior object models or maps?
• RQ2 How robust is the system to natural environmental variations such as lighting changes, clutter, and object repositioning?
• RQ3 To what extent can learned visual embeddings enable accurate behavior execution across different scene configurations and viewpoints?
• RQ4 How effective is the task graph architecture in enabling reuse and recovery during execution of complex, multi-behavior tasks?
• RQ5 What is the end-to-end success rate of the system in real-world homes under diverse and unstructured conditions?

Key findings

• The system achieved an overall success rate of 85% across 60 end-to-end task executions in two real homes, with three complex tasks averaging 45 behaviors each.
• The system demonstrated robustness to significant environmental variations, including lighting changes, added obstacles, repositioned objects, and cabinet door state changes (open, closed, partially open).
• Behavior failures occurred in only 0.4% of cases, primarily due to inaccurate object pose estimation or incorrect keyframe matching (e.g., misclassifying a partially open cabinet as closed).
• The fault detection and recovery mechanism enabled 99.6% of behaviors to succeed or recover, with no catastrophic failures observed during testing.
• The robot performed tasks 10 to 100 times slower than a human (average 20x slower), indicating room for performance optimization.
• The use of learned dense visual embeddings enabled reliable behavior execution across diverse visual conditions, including viewpoint changes and scene clutter, without requiring explicit object recognition or segmentation.