[Paper Review] Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion

Zipeng Fu, Xuxin Cheng+1 more|arXiv (Cornell University)|Oct 18, 2022

Muscle activation and electromyography studies25 citations

TL;DR

The authors learn a single unified policy to coordinate a legged quadruped and an attached arm for simultaneous manipulation and locomotion, with a Regularized Online Adaptation module to bridge Sim-to-Real and Advantage Mixing to speed training.

ABSTRACT

An attached arm can significantly increase the applicability of legged robots to several mobile manipulation tasks that are not possible for the wheeled or tracked counterparts. The standard hierarchical control pipeline for such legged manipulators is to decouple the controller into that of manipulation and locomotion. However, this is ineffective. It requires immense engineering to support coordination between the arm and legs, and error can propagate across modules causing non-smooth unnatural motions. It is also biological implausible given evidence for strong motor synergies across limbs. In this work, we propose to learn a unified policy for whole-body control of a legged manipulator using reinforcement learning. We propose Regularized Online Adaptation to bridge the Sim2Real gap for high-DoF control, and Advantage Mixing exploiting the causal dependency in the action space to overcome local minima during training the whole-body system. We also present a simple design for a low-cost legged manipulator, and find that our unified policy can demonstrate dynamic and agile behaviors across several task setups. Videos are at https://maniploco.github.io

Motivation & Objective

• Motivate enabling mobile manipulation on legged robots by tightly coordinating arm and leg control.
• Develop a single, end-to-end policy that unifies manipulation and locomotion.
• Address sim-to-real transfer without a two-stage teacher-student setup.
• Demonstrate robust learning via a low-cost hardware platform and diverse task setups.

Proposed method

• Formulate a single neural policy pi that takes base, arm, and leg states plus previous actions and environment extrinsics as input and outputs arm and leg target joint positions.
• Train with reinforcement learning using a combined manipulation and locomotion reward by PPO.
• Introduce Advantage Mixing to decompose credit assignment by mixing manipulation and locomotion advantages during policy updates.
• Propose Regularized Online Adaptation to bridge Sim-to-Real by learning an environment extrinsics latent z_mu from privileged simulation data and regularizing it toward z_phi inferred from onboard observations.
• Use joint-space position control with PD torques for arms and legs to simplify learning and reduce Sim-to-Real gap.
• Provide a low-cost, untethered hardware platform (Go1 quadruped with a WidowX arm) for real-world evaluation.

Experimental results

Research questions

• RQ1 Can a single unified policy coordinate legged locomotion and arm manipulation more effectively than decoupled or partially coupled controllers?
• RQ2 Does Advantage Mixing accelerate learning and improve credit assignment for simultaneous manipulation and locomotion?
• RQ3 Can Regularized Online Adaptation provide robust sim-to-real transfer without a two-phase teacher-student pipeline?

Key findings

• The unified policy outperforms separate and uncoordinated baselines across multiple metrics, achieving higher survival and comparable or lower energy use.
• Advantage Mixing accelerates learning and improves command-following for both manipulation and locomotion, reducing convergence time.
• Regularized Online Adaptation yields better sim-to-real transfer than Rapid Motor Adaptation and Domain Randomization, with smaller imitation error and improved EE tracking.
• The unified policy increases the arm workspace and improves stability under perturbations, indicating strong whole-body coordination between legs and arm.
• Real-world experiments show agile, coordinated leg-arm motion and superior task success and speed compared to a baseline MPC+IK controller.