[Paper Review] Learning-from-Observation System Considering Hardware-Level Reusability

Jun Takamatsu, Kazuhiro Sasabuchi+3 more|arXiv (Cornell University)|Dec 19, 2022

Robotic Path Planning Algorithms4 citations

TL;DR

This paper proposes a learning-from-observation system that enables hardware-level reusability by representing tasks in robot-agnostic form and decoupling hand-specific skills from robot body control. By using a generic inverse kinematics solver and a skill library focused solely on the robot hand, the same task sequences and parameters were successfully executed on two distinct robots—Nextage and Fetch—demonstrating high reusability across different hardware platforms with minimal reconfiguration.

ABSTRACT

Robot developers develop various types of robots for satisfying users' various demands. Users' demands are related to their backgrounds and robots suitable for users may vary. If a certain developer would offer a robot that is different from the usual to a user, the robot-specific software has to be changed. On the other hand, robot-software developers would like to reuse their developed software as much as possible to reduce their efforts. We propose the system design considering hardware-level reusability. For this purpose, we begin with the learning-from-observation framework. This framework represents a target task in robot-agnostic representation, and thus the represented task description can be shared with various robots. When executing the task, it is necessary to convert the robot-agnostic description into commands of a target robot. To increase the reusability, first, we implement the skill library, robot motion primitives, only considering a robot hand and we regarded that a robot was just a carrier to move the hand on the target trajectory. The skill library is reusable if we would like to the same robot hand. Second, we employ the generic IK solver to quickly swap a robot. We verify the hardware-level reusability by applying two task descriptions to two different robots, Nextage and Fetch.

Motivation & Objective

• To address the conflict between custom robot development and software reusability in robotics.
• To enable robot-agnostic task representation that can be reused across different robot hardware platforms.
• To reduce the effort required to adapt software for new robots by isolating hand-specific skills and decoupling body motion control.
• To validate hardware-level reusability through cross-platform execution on Nextage and Fetch robots.

Proposed method

• Representing tasks as symbolic sequences (e.g., grasp, pick, place) in a robot-agnostic format derived from human demonstrations.
• Implementing a skill library focused only on the robot hand, with reinforcement learning used to train grasp and trajectory-following behaviors.
• Using a generic inverse kinematics solver and body-role division to abstract robot body motion control, enabling easy robot swapping.
• Estimating skill parameters (e.g., grasp pose) from depth images using a CNN trained on superquadric object models and RANSAC-based segmentation.
• Detecting hitting events via force feedback from finger joints to trigger task transitions, such as terminating trajectory generation.
• Mapping the robot-agnostic task sequence to target robot commands through a modular converter that handles hardware-specific mappings.

Experimental results

Research questions

• RQ1 Can task representations learned from human demonstrations be reused across different robot hardware platforms?
• RQ2 To what extent can skill libraries be shared when only the robot hand remains consistent across platforms?
• RQ3 How effective is a generic inverse kinematics solver in enabling rapid robot swapping without re-implementing motion control?
• RQ4 Can collision avoidance be effectively taught through demonstration and generalized across different robots using shared trajectory parameters?

Key findings

• The same task sequences—place-on-plate and shelf demos—were successfully executed on both Nextage and Fetch robots using the same skill library and parameters.
• The system achieved hardware-level reusability by reusing the skill library across robots with different kinematics and physical forms, provided the hand (Shadow Hand Lite) remained the same.
• The use of a generic IK solver enabled seamless robot switching with minimal changes to the control stack.
• Force feedback from the hand’s joints successfully detected hitting events, allowing consistent task transition logic across robots.
• The CNN-based parameter estimation from depth images enabled accurate grasp pose prediction without requiring robot-specific calibration.
• The system demonstrated robustness to minor environmental differences, with sensor feedback enabling adaptation during execution.