[Paper Review] Human-in-the-loop Robotic Manipulation Planning for Collaborative Assembly

Mohamed Raessa, Jimmy Chi Yin Chen+2 more|arXiv (Cornell University)|Sep 25, 2019

Robot Manipulation and Learning39 references4 citations

TL;DR

This paper proposes a human-in-the-loop robotic manipulation planner for collaborative cabinet assembly, where a dual-arm robot handles pick-and-place tasks with constrained motion planning and ergonomic handover, while humans perform fine operations like screwing and alignment. The system reduces human physical workload and improves efficiency by optimizing object orientation and robot motion for safe, comfortable collaboration.

ABSTRACT

This paper develops a robotic manipulation planner for human-robot collaborative assembly. Unlike previous methods which study an independent and fully AI-equipped autonomous system, this paper explores the subtask distribution between a robot and a human and studies a human-in-the-loop robotic system for collaborative assembly. The system distributes the subtasks of an assembly to robots and humans by exploiting their advantages and avoiding their disadvantages. The robot in the system will work on pick-and-place tasks and provide workpieces to humans. The human collaborator will work on fine operations like aligning, fixing, screwing, etc. A constraint based incremental manipulation planning method is proposed to generate the motion for the robots. The performance of the proposed system is demonstrated by asking a human and the dual-arm robot to collaboratively assemble a cabinet. The results showed that the proposed system and planner are effective, efficient, and can assist humans in finishing the assembly task comfortably.

Motivation & Objective

• Address the limitations of fully autonomous robots in handling fine manipulation tasks in assembly.
• Reduce human physical workload in labor-intensive assembly processes by offloading repetitive pick-and-place tasks to robots.
• Improve collaboration efficiency by distributing subtasks based on human and robot strengths: robots for transport and positioning, humans for precision tasks.
• Enhance user comfort and safety by incorporating human ergonomics into robotic motion planning and handover pose selection.
• Develop a constraint-based incremental planning method that ensures stable manipulation of thin and reoriented workpieces using soft finger contact constraints.

Proposed method

• Utilizes a dual-arm robot with parallel grippers and suction cup tools to handle various workpieces, including thin boards.
• Employs RGB cameras and AR markers for real-time workpiece recognition and pose estimation.
• Applies a constraint-based incremental manipulation planning method that enforces soft finger contact constraints to prevent slipping during grasping and regrasping.
• Optimizes robot motion trajectories to minimize human physical strain by aligning handover poses with the human’s natural posture and reach.
• Integrates FT sensors at end-effectors to monitor contact forces and ensure safe manipulation during handover and assembly.
• Uses a workflow where the robot prepares the next workpiece while the human completes the current subtask, enabling continuous collaboration.

Experimental results

Research questions

• RQ1 How can subtasks in collaborative assembly be effectively distributed between humans and robots to leverage their respective strengths?
• RQ2 What constraints and planning strategies are necessary to ensure safe and stable manipulation of thin, reorientable workpieces during robot-assisted handover?
• RQ3 To what extent does incorporating human ergonomics into robotic motion planning reduce physical workload during collaborative assembly?
• RQ4 How does the proposed human-in-the-loop system compare to fully autonomous robots or human-only assembly in terms of efficiency and user comfort?
• RQ5 Can incremental, constraint-based motion planning enable reliable and safe manipulation in dynamic, real-world collaborative environments?

Key findings

• The proposed system significantly reduces human physical workload, eliminating the need for forward torso bending and decreasing shoulder strain by presenting boards in optimized, already-flipped poses.
• Human reaching and picking actions became more comfortable due to improved handover positioning, with reduced arm movement and better alignment with natural posture.
• The idle time for the human increased as the robot assumed preparatory tasks, leading to smoother workflow continuity and reduced task-switching fatigue.
• The constraint-based motion planner successfully enabled stable manipulation of thin boards through soft finger contact constraints, preventing slippage during reorientation and handover.
• Experiments showed that the human-robot collaboration system outperformed both human-only and robot-only approaches in terms of comfort, efficiency, and task completion speed.
• The system demonstrated robust performance even in constrained workspaces, such as during the final stages of cabinet assembly, where workspace clearance was limited.