[Paper Review] Efficient Skill Acquisition for Complex Manipulation Tasks in Obstructed Environments

Jun Yamada, Jack Collins+1 more|arXiv (Cornell University)|Mar 6, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a modular framework that combines object-centric generative models (OCGM) for one-shot goal identification, motion planning (MP) for obstacle-avoiding navigation, and a reinforcement learning (RL) policy with a skill transition network for complex manipulation in obstructed environments. The system achieves 86.7–96.7% success rates on real-world industrial insertion tasks using only a few demonstrations, outperforming state-of-the-art baselines by a significant margin.

ABSTRACT

Data efficiency in robotic skill acquisition is crucial for operating robots in varied small-batch assembly settings. To operate in such environments, robots must have robust obstacle avoidance and versatile goal conditioning acquired from only a few simple demonstrations. Existing approaches, however, fall short of these requirements. Deep reinforcement learning (RL) enables a robot to learn complex manipulation tasks but is often limited to small task spaces in the real world due to sample inefficiency and safety concerns. Motion planning (MP) can generate collision-free paths in obstructed environments, but cannot solve complex manipulation tasks and requires goal states often specified by a user or object-specific pose estimator. In this work, we propose a system for efficient skill acquisition that leverages an object-centric generative model (OCGM) for versatile goal identification to specify a goal for MP combined with RL to solve complex manipulation tasks in obstructed environments. Specifically, OCGM enables one-shot target object identification and re-identification in new scenes, allowing MP to guide the robot to the target object while avoiding obstacles. This is combined with a skill transition network, which bridges the gap between terminal states of MP and feasible start states of a sample-efficient RL policy. The experiments demonstrate that our OCGM-based one-shot goal identification provides competitive accuracy to other baseline approaches and that our modular framework outperforms competitive baselines, including a state-of-the-art RL algorithm, by a significant margin for complex manipulation tasks in obstructed environments.

Motivation & Objective

• To address the challenge of data-efficient skill acquisition in small-batch manufacturing settings with arbitrary obstacles.
• To overcome the limitations of motion planning (MP) requiring explicit goal states and reinforcement learning (RL) suffering from sample inefficiency and safety issues in real-world settings.
• To enable one-shot target object identification and re-identification in new scenes without fine-tuning, using an object-centric generative model (OCGM).
• To bridge the gap between terminal states of MP and feasible start states of RL policies through a skill transition network.
• To demonstrate superior performance in complex manipulation tasks in obstructed environments compared to state-of-the-art baselines, including sample-efficient RL and heuristic-based methods.

Proposed method

• The system uses a pre-trained object-centric generative model (OCGM) to extract and re-identify the target object from a single human demonstration, enabling one-shot goal specification.
• The OCGM infers the 3D position of the target object in new scenes by matching its object-centric representation, providing a goal pose for motion planning.
• Motion planning generates a collision-free trajectory to the target object while avoiding obstacles, using the OCGM's output as the goal.
• A skill transition network is introduced to map the terminal state of MP to a feasible starting state for the RL policy, reducing failure during policy initiation.
• A sample-efficient RL policy is trained from sparse rewards using only a few demonstrations to guide exploration, avoiding the need for dense reward engineering.
• The modular pipeline integrates OCGM-based goal identification, MP for safe navigation, and RL with transition network for task completion, enabling robust performance in real-world obstructed environments.

Figure 1 : Task setup. We solve complex manipulation tasks within the entire operational space of a robot by leveraging an OCGM for versatile and efficient goal acquisition paired with MP and RL. Note that obstacles and a socket are randomly placed on the table.

Experimental results

Research questions

• RQ1 Can an object-centric generative model (OCGM) enable accurate and versatile one-shot target object identification and re-identification in new scenes without object-specific fine-tuning?
• RQ2 How effective is the proposed OCGM-based goal specification method compared to pose estimation baselines in terms of accuracy and robustness?
• RQ3 Does the integration of motion planning and reinforcement learning with a skill transition network significantly improve success rates in complex manipulation tasks within obstructed environments?
• RQ4 To what extent does the skill transition network reduce failures caused by mismatched initial states between motion planning and RL policy execution?
• RQ5 Can the proposed modular framework outperform state-of-the-art RL algorithms and alternative MP+RL baselines in real-world industrial manipulation tasks?

Key findings

• The OCGM-based one-shot goal identification method achieves comparable accuracy to other baseline approaches in target object re-identification across new scenes.
• The proposed system (MP+RL with skill transition network) achieves a success rate of 86.7% on VGA, 83.3% on RJ45, 93.3% on E-model, and 96.7% on USB-A tasks, significantly outperforming all baselines.
• The method without the skill transition network (MP+RL w/o skill transition) achieves 73.3–80.0% success rates, indicating the transition network is critical for high performance.
• The SAC baseline fails entirely (0.0% success rate) on all tasks, highlighting the challenge of sample efficiency in real-world RL with obstructions.
• The main failure mode is misidentification of the target object by the OCGM, suggesting that model robustness can be further improved with extended training data.
• The method outperforms MP+Demonstration Replay (3.3–26.7% success), MP+BC (16.7–26.7%), and MP+Heuristic (10.0–43.3%) across all tasks, demonstrating its superiority in real-world applicability.

Figure 2 : Our framework architecture. (a) We leverage an OCGM to re-identify a target object such that its object-centric representation matches one extracted from a single demonstration. The goal state is specified in the robot’s reference frame using an external RGB-D camera with calibrated extri