[Paper Review] Residual Policy Learning

Tom Silver, Kelsey R. Allen+2 more|arXiv (Cornell University)|Dec 15, 2018

Reinforcement Learning in RoboticsComputer Science40 references52 citations

TL;DR

RPL augments arbitrary initial policies with a learnable residual using deep RL, improving performance and data efficiency on long-horizon, sparse-reward robotic manipulation tasks.

ABSTRACT

We present Residual Policy Learning (RPL): a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch remains data-inefficient or intractable, but learning a residual on top of the initial controller can yield substantial improvements. We study RPL in six challenging MuJoCo tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration. For initial controllers, we consider both hand-designed policies and model-predictive controllers with known or learned transition models. By combining learning with control algorithms, RPL can perform long-horizon, sparse-reward tasks for which reinforcement learning alone fails. Moreover, we find that RPL consistently and substantially improves on the initial controllers. We argue that RPL is a promising approach for combining the complementary strengths of deep reinforcement learning and robotic control, pushing the boundaries of what either can achieve independently. Video and code at https://k-r-allen.github.io/residual-policy-learning/.

Motivation & Objective

• Motivate and address the data inefficiency of learning from scratch in complex robotic manipulation tasks.
• Propose a simple residual learning framework to augment any initial policy.
• Show that learning a residual improves performance across varied sources of initial policies.
• Demonstrate data efficiency gains and robustness in challenging MuJoCo tasks with partial observability, noise, and model mis-specification.

Proposed method

• Augment an initial policy pi with a learned residual f_theta to form pi_theta(s) = pi(s) + f_theta(s).
• Treat the residual as a policy in a residual MDP M^(pi) with transition T^(pi)(s,a,s') = T(s, pi(s) + a, s').
• Use model-free deep RL methods (DDPG with HER) to learn f_theta, with last layer initialized to zero to preserve pi.
• Optionally fix the critic during a burn-in period to stabilize learning when the initial policy is strong but the critic lags.
• Extend to recurrent policies for POMDPs via short state histories when needed.
• Compare against baselines including the initial policy alone, learning from scratch (DDPG+HER), and an Expert-Explore variant to isolate exploration benefits.

Experimental results

Research questions

• RQ1 Can residual policy learning consistently improve upon a variety of initial policies (hand-designed, MPC, or cached model-based controllers)?
• RQ2 Is RPL more data-efficient than learning from scratch in long-horizon, sparse-reward robotic tasks?
• RQ3 How does RPL perform under partial observability, sensor noise, and model misspecification?
• RQ4 Do residual policies preserve the performance of good initial policies while correcting their failures?
• RQ5 Can RPL beat or complement model-based RL methods in practice?

Key findings

• RPL substantially improves on initial policies across six MuJoCo manipulation tasks.
• In many cases, RPL converges with far fewer samples than learning from scratch (e.g., roughly 10x fewer in PickAndPlace).
• RPL demonstrates robustness to sensor noise and structured uncertainty, achieving high success rates where baselines fail.
• RPL can outperform model-based RL baselines (PETS) and can accelerate convergence when used on top of model-based controllers (CachedPETS).
• The Expert-Explore baseline shows improved exploration but does not fully account for RPL’s data efficiency, indicating benefits from residual parameterization and initialization.