[Paper Review] A Learning Framework for High Precision Industrial Assembly

Yongxiang Fan, Jieliang Luo+1 more|arXiv (Cornell University)|Sep 23, 2018

Reinforcement Learning in Robotics13 references4 citations

TL;DR

This paper proposes a hybrid learning framework that combines model-based trajectory optimization with actor-critic reinforcement learning to enable high-precision industrial assembly. By using optimal trajectories as a semi-supervisor and a critic network to stabilize policy learning, the method achieves faster convergence and greater robustness than pure reinforcement learning, reducing data requirements and improving stability in force/torque feedback environments with high-stiffness systems.

ABSTRACT

Automatic assembly has broad applications in industries. Traditional assembly tasks utilize predefined trajectories or tuned force control parameters, which make the automatic assembly time-consuming, difficult to generalize, and not robust to uncertainties. In this paper, we propose a learning framework for high precision industrial assembly. The framework combines both the supervised learning and the reinforcement learning. The supervised learning utilizes trajectory optimization to provide the initial guidance to the policy, while the reinforcement learning utilizes actor-critic algorithm to establish the evaluation system even the supervisor is not accurate. The proposed learning framework is more efficient compared with the reinforcement learning and achieves better stability performance than the supervised learning. The effectiveness of the method is verified by both the simulation and experiment.

Motivation & Objective

• To address the limitations of traditional assembly methods that rely on manual trajectory tuning and are sensitive to environmental uncertainties.
• To improve data efficiency and training stability in high-precision assembly tasks involving force/torque feedback and rigid robot dynamics.
• To develop a learning framework that combines the strengths of model-based optimization and model-free reinforcement learning for better performance and reliability.
• To enable robust generalization across varying hole positions, shapes, and clearances in peg-in-hole and similar tasks.

Proposed method

• The framework uses trajectory optimization with position and force/torque feedback as a semi-supervisor to generate initial optimal trajectories.
• An actor-critic network is trained using both supervised learning from the optimal trajectories and policy gradient updates from the critic network.
• The critic network learns Q-values based on both the supervisor’s output and real-time policy exploration, providing a stable ground-truth evaluation of policy quality.
• The actor network is updated via a combination of supervised guidance and policy gradient signals, improving convergence and stability.
• A hybrid training strategy reduces reliance on random exploration, narrowing the exploration space and enhancing data efficiency.
• The method integrates a low-level tracking controller to execute velocity commands derived from the policy, ensuring accurate end-effector motion.

Experimental results

Research questions

• RQ1 Can a hybrid learning framework combining model-based optimization and actor-critic reinforcement learning improve data efficiency and stability in high-precision assembly?
• RQ2 How does the integration of a critic network mitigate instability caused by rigid robot dynamics and force/torque feedback in trajectory optimization?
• RQ3 To what extent can the learned policy generalize across variations in hole position, shape, and clearance?
• RQ4 How does the proposed framework compare in sample efficiency and convergence speed to pure DDPG, TD3, and SAC in simulation and real-world experiments?

Key findings

• The guided-DDPG framework achieved success in the Lego brick insertion task at 800 rollouts, while pure DDPG required 5,000 rollouts and showed inconsistent performance.
• The proposed method reduced training time from 83 minutes (DDPG) to 37.3 minutes and rollouts from 7,000 to 1,500, demonstrating significant improvements in data and time efficiency.
• The policy successfully inserted bricks with 0.5 mm hole position uncertainty, 5 mm maximum offset, and various shapes including 4×2 bricks and cylinders with incomplete holes.
• The policy generalized across clearance variations, successfully inserting into holes with 0.1 mm, 1 μm, and 0 μm clearances, demonstrating robustness to geometric uncertainty.
• In experiments, guided-DDPG found a policy in a 3 mm exploration radius in 1.5 hours, while pure DDPG required 2 hours to find a policy in a 1 mm radius, indicating broader and more efficient exploration.
• The critic network effectively stabilized training by correcting for inaccuracies in the trajectory optimizer, preventing policy collapse and improving reliability.