[Paper Review] Vision-based Robot Manipulation Learning via Human Demonstrations

Zhixin Jia, Mengxiang Lin+2 more|arXiv (Cornell University)|Mar 1, 2020

Multimodal Machine Learning Applications22 references7 citations

TL;DR

This paper proposes a vision-based robot manipulation learning framework that enables generalization across objects and environments using only a single third-person demonstration. By combining action recognition, object detection, and a text-based prior knowledge base for action-object reasoning, the system achieves over 90% success rate in real-world execution of simple and complex daily tasks with novel objects and varying contexts.

ABSTRACT

Vision-based learning methods provide promise for robots to learn complex manipulation tasks. However, how to generalize the learned manipulation skills to real-world interactions remains an open question. In this work, we study robotic manipulation skill learning from a single third-person view demonstration by using activity recognition and object detection in computer vision. To facilitate generalization across objects and environments, we propose to use a prior knowledge base in the form of a text corpus to infer the object to be interacted with in the context of a robot. We evaluate our approach in a real-world robot, using several simple and complex manipulation tasks commonly performed in daily life. The experimental results show that our approach achieves good generalization performance even from small amounts of training data.

Motivation & Objective

• To enable robots to learn complex manipulation skills from a single third-person visual demonstration without requiring extensive real-world or simulated training data.
• To improve generalization across diverse objects and environments by incorporating prior knowledge about action-object relationships.
• To develop a scalable, real-world applicable imitation-from-observation framework that avoids the need for task-specific data collection on real robots.
• To validate the effectiveness of the approach on a range of daily-life manipulation tasks under varying object and environmental conditions.

Proposed method

• A deep convolutional neural network (AP-CNN) is used to classify action primitives from video sequences of human demonstrations.
• A window filter algorithm extracts key action primitives from the continuous action sequence to improve temporal modeling.
• Mask R-CNN performs instance segmentation and object detection to identify object identities and 3D poses in the robot's visual field.
• Principal Component Analysis (PCA) transforms segmented masks into physical quantities for robot control.
• A Bayesian probability model integrates a manually curated text corpus of daily-life action-object pairs to infer the correct object for interaction during task execution.
• The action planner uses the prior knowledge base to reason about which object should be manipulated based on the demonstrated action and environmental context.

Experimental results

Research questions

• RQ1 Can a robot generalize manipulation skills to novel objects and environments using only a single third-person demonstration?
• RQ2 How effective is the integration of a text-based prior knowledge base in improving object inference during manipulation planning?
• RQ3 To what extent can vision-based action recognition and object detection enable robust skill transfer in real-world robotic manipulation?
• RQ4 What is the performance of the system on complex, multi-step manipulation tasks involving unseen objects?

Key findings

• The system achieved a success rate of 100% on five simple manipulation tasks (pick-place, push away, deliver object) when tested with novel objects and randomized object placements.
• For the open-bottle and pour-water tasks, the success rate was 90% (9 out of 10 trials), demonstrating effective generalization despite object variations.
• In complex tasks involving multiple sub-tasks (e.g., clearing a table and delivering a box, opening a bottle and pouring into a container), the system achieved a 90% success rate (9 out of 10 trials) with unseen objects.
• Failures were primarily attributed to object detection errors or inconsistent action primitive extraction, indicating room for improvement in perception and action sequence modeling.
• The framework demonstrated robust performance across diverse real-world scenarios without requiring additional real-robot or simulation data for training beyond initial perception models.