[Paper Review] Robotic Imitation of Human Assembly Skills Using Hybrid Trajectory and Force Learning

Yan Wang, Cristian C. Beltran-Hernandez+2 more|arXiv (Cornell University)|Mar 10, 2021

Robot Manipulation and Learning32 references4 citations

TL;DR

This paper proposes a hybrid learning framework that combines hierarchical goal-conditioned imitation learning (HGCIL) for trajectory acquisition and reinforcement learning-based force control to enable robots to imitate human assembly skills. The method achieves robust, high-quality trajectory and force control in real-world assembly tasks with low-clearance insertions and variable force requirements, outperforming baseline methods in sample efficiency and real-world transferability.

ABSTRACT

Robotic assembly tasks involve complex and low-clearance insertion trajectories with varying contact forces at different stages. While the nominal motion trajectory can be easily obtained from human demonstrations through kinesthetic teaching, teleoperation, simulation, among other methods, the force profile is harder to obtain especially when a real robot is unavailable. It is difficult to obtain a realistic force profile in simulation even with physics engines. Such simulated force profiles tend to be unsuitable for the actual robotic assembly due to the reality gap and uncertainty in the assembly process. To address this problem, we present a combined learning-based framework to imitate human assembly skills through hybrid trajectory learning and force learning. The main contribution of this work is the development of a framework that combines hierarchical imitation learning, to learn the nominal motion trajectory, with a reinforcement learning-based force control scheme to learn an optimal force control policy, that can satisfy the nominal trajectory while adapting to the force requirement of the assembly task. To further improve the imitation learning part, we develop a hierarchical architecture, following the idea of goal-conditioned imitation learning, to generate the trajectory learning policy on the extit{skill} level offline. Through experimental validations, we corroborate that the proposed learning-based framework is robust to uncertainty in the assembly task, can generate high-quality trajectories, and can find suitable force control policies, which adapt to the task's force requirements more efficiently.

Motivation & Objective

• To address the challenge of learning realistic force profiles in robotic assembly tasks due to the reality gap in simulation.
• To develop a robust framework that combines trajectory and force learning for low-clearance, contact-rich assembly tasks.
• To improve sample efficiency and generalization by using hierarchical imitation learning for trajectory policy learning.
• To enable real-world transfer of policies trained in simulation with minimal fine-tuning.
• To adapt force control dynamically to varying task phases (search vs. insertion) under uncertainty.

Proposed method

• The framework uses hierarchical goal-conditioned imitation learning (HGCIL) to learn a high-level skill policy that generates sub-goals for trajectory execution, improving sample efficiency and robustness.
• A parallel position/force controller combines PD and PI control: PD regulates position trajectory based on pose error, while PI adjusts force based on measured contact force.
• The force control policy is learned via reinforcement learning (RL), with the policy trained to maintain desired force levels while following the nominal trajectory.
• The system uses a two-stage training process: offline HGCIL for trajectory policy, followed by RL-based force policy training with transfer learning from prior tasks.
• The controller dynamically adjusts gains $K_p^p$ and $K_p^f$ based on task phase—increasing $K_p^p$ during insertion for stability and reducing $K_p^f$ to limit force.
• Demonstration data is collected via kinesthetic teaching or simulation, and the force policy is initialized using a pre-trained policy from a similar task to accelerate convergence.

Experimental results

Research questions

• RQ1 Can a hierarchical imitation learning approach effectively learn complex, low-clearance insertion trajectories from human demonstrations with improved sample efficiency?
• RQ2 How can a reinforcement learning-based force control policy adapt to varying force requirements across different phases of an assembly task?
• RQ3 To what extent can a policy trained in simulation be successfully transferred to real-world robotic hardware without extensive re-tuning?
• RQ4 How does the hybrid trajectory-force learning framework handle uncertainty in trajectory and force during real-world assembly?
• RQ5 Can the proposed framework outperform standard behavioral cloning and goal-conditioned behavioral cloning with relabeling in terms of task success and sample efficiency?

Key findings

• The HGCIL method achieved higher-quality trajectory imitation and better sample efficiency compared to vanilla behavioral cloning and GCBC-relabeling baselines in simulation.
• The policy trained on simulation (P-14 model) successfully transferred to a real UR3e robot without retraining, completing the L-insertion task in 33.3 minutes of training.
• In the HDMI insertion task with obstacles and friction, the learned policy completed the task in 16.3 seconds (326 steps), while the initial policy failed within 25 seconds.
• The controller dynamically adjusted $K_p^p$ and $K_p^f$ during execution: increasing $K_p^p$ during insertion for trajectory adherence and reducing $K_p^f$ to prevent force overshoot.
• The framework demonstrated robustness to trajectory uncertainty and force uncertainty, maintaining stable contact forces across varying insertion phases.
• Transfer learning using a pre-trained force policy from the L-insertion task significantly reduced training time for subsequent tasks like condenser and HDMI assembly.