[Paper Review] Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space

Maximilian Ulmer, Elie Aljalbout+2 more|arXiv (Cornell University)|Oct 19, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes AFORCE, a bio-inspired hierarchical action space that combines reinforcement learning (RL) with real-time adaptive force-impedance control to improve sample efficiency, energy efficiency, and safety in robotic manipulation. By decoupling high-level task planning from low-level physical interaction control, AFORCE enables stable, compliant manipulation on real hardware and outperforms baseline action spaces in simulation and real-world contact-rich tasks.

ABSTRACT

Intelligent agents must be able to think fast and slow to perform elaborate manipulation tasks. Reinforcement Learning (RL) has led to many promising results on a range of challenging decision-making tasks. However, in real-world robotics, these methods still struggle, as they require large amounts of expensive interactions and have slow feedback loops. On the other hand, fast human-like adaptive control methods can optimize complex robotic interactions, yet fail to integrate multimodal feedback needed for unstructured tasks. In this work, we propose to factor the learning problem in a hierarchical learning and adaption architecture to get the best of both worlds. The framework consists of two components, a slow reinforcement learning policy optimizing the task strategy given multimodal observations, and a fast, real-time adaptive control policy continuously optimizing the motion, stability, and effort of the manipulator. We combine these components through a bio-inspired action space that we call AFORCE. We demonstrate the new action space on a contact-rich manipulation task on real hardware and evaluate its performance on three simulated manipulation tasks. Our experiments show that AFORCE drastically improves sample efficiency while reducing energy consumption and improving safety.

Motivation & Objective

• To address the sample inefficiency and instability of deep reinforcement learning in real-world robotic manipulation tasks.
• To integrate fast, reactive adaptive control with slow, strategic RL policy learning for improved performance.
• To develop a novel action space that enables continuous adaptation of force and impedance during contact-rich tasks.
• To evaluate the method in simulation and on real hardware, focusing on sample efficiency, energy consumption, and safety.
• To demonstrate that hierarchical control with bio-inspired action spaces can outperform standard RL action spaces in complex manipulation.

Proposed method

• AFORCE introduces a Cartesian adaptive force-impedance control action space that decouples high-level task policy from low-level physical interaction control.
• The high-level RL policy operates on a small, multimodal observation space to plan task strategies, while the low-level adaptive controller continuously adjusts force and impedance for stability and compliance.
• The framework uses a hierarchical control architecture inspired by human motor control, where the RL policy sets desired force and impedance parameters, and the adaptive controller enacts them in real time.
• The action space is designed to support continuous, reactive adaptation of mechanical impedance and contact force, crucial for disturbance rejection and energy efficiency.
• The method is evaluated in simulation on three tasks (Lift, Wipe, Door) and validated on real hardware using a Franka Emika Panda robot.
• Adaptive parameters such as stiffness and damping are manually tuned, though future work aims to learn them jointly with the policy.

Experimental results

Research questions

• RQ1 Can a bio-inspired hierarchical action space improve sample efficiency in deep reinforcement learning for robotic manipulation?
• RQ2 How does AFORCE compare to standard action spaces in terms of learning stability and convergence speed?
• RQ3 To what extent does AFORCE improve energy efficiency and safety in contact-rich manipulation tasks?
• RQ4 Can the integration of real-time adaptive control with RL lead to stable, compliant behavior on real hardware?
• RQ5 Does the separation of task planning from physical interaction control enhance robustness and performance?

Key findings

• AFORCE achieved consistent maximum reward in the Wipe task after only 500,000 training steps, outperforming all baselines in learning speed and success rate.
• In the Wipe task, AFORCE successfully removed all dirt markers after just a few training episodes, while other methods failed to maintain consistent contact or force control.
• AFORCE reduced average energy consumption compared to variable and variable+force action spaces, particularly in the Wipe environment.
• AFORCE achieved the lowest penalty rate among all methods, with only 12% of episodes receiving a safety penalty in the Wipe task, compared to over 75% for variable-based action spaces.
• The low+force action space had the lowest average penalty per episode (17.16), but AFORCE came close (17.76), while variable spaces averaged over 400 penalties per episode.
• AFORCE demonstrated stable learning with low reward variance across multiple random seeds, indicating robust policy convergence.