[Paper Review] Online Tool and Task learning via Human Robot Interaction.

Masood Dehghan, Zichen Vincent Zhang+4 more|arXiv (Cornell University)|Sep 24, 2018

Robot Manipulation and Learning20 references3 citations

TL;DR

This paper presents a robotic system that incrementally learns new tools and tasks through real-time human interaction using a deep learning-based object recognition module, an intuitive 3D motion specification interface, and a hybrid force-vision control system for compliant manipulation on unstructured surfaces. The system was recognized as a finalist in the KUKA Innovation Award and demonstrated at Hanover Messe 2018.

ABSTRACT

This work describes the development of a robotic system that acquires knowledge incrementally through human interaction where new tools and motions are taught on the fly. The robotic system developed was one of the five finalists in the KUKA Innovation Award competition and demonstrated during the Hanover Messe 2018 in Germany. The main contributions of the system are a) a novel incremental object learning module - a deep learning based localization and recognition system - that allows a human to teach new objects to the robot, b) an intuitive user interface for specifying 3D motion task associated with the new object, c) a hybrid force-vision control module for performing compliant motion on an unstructured surface. This paper describes the implementation and integration of the main modules of the system and summarizes the lessons learned from the competition.

Motivation & Objective

• To enable robots to learn new tools and motion tasks incrementally through direct human interaction in real time.
• To address the challenge of teaching robots new objects and actions without pre-programmed knowledge or structured environments.
• To develop an intuitive interface for specifying 3D motion tasks associated with newly taught objects.
• To enable compliant manipulation on unstructured surfaces using a hybrid force-vision control strategy.

Proposed method

• A deep learning-based incremental object learning module for real-time localization and recognition of new tools and objects during interaction.
• An intuitive user interface allowing humans to define 3D motion tasks associated with newly taught objects through direct input.
• A hybrid force-vision control module that combines visual feedback with force sensing to enable compliant motion on unstructured surfaces.
• Integration of the three core modules—object learning, task specification, and compliant control—into a unified robotic system for online learning.
• System deployment and validation through participation in the KUKA Innovation Award competition and live demonstration at Hanover Messe 2018.

Experimental results

Research questions

• RQ1 How can a robot incrementally learn new tools and objects through continuous human interaction in real time?
• RQ2 What interface design enables intuitive, 3D motion task specification for newly taught objects?
• RQ3 How can a robot perform compliant manipulation on unstructured surfaces using real-time visual and force feedback?
• RQ4 What are the key system-level integration challenges in enabling online tool and task learning?

Key findings

• The deep learning-based object learning module successfully enabled real-time localization and recognition of new tools and objects during interaction.
• The intuitive user interface allowed non-expert users to specify 3D motion tasks for new objects with minimal training.
• The hybrid force-vision control module enabled stable and compliant manipulation on unstructured surfaces, improving task robustness.
• The integrated system was successfully demonstrated at Hanover Messe 2018 and selected as one of five finalists in the KUKA Innovation Award competition.