[Paper Review] Control Regularization for Reduced Variance Reinforcement Learning

Richard Cheng, Abhinav Verma+4 more|arXiv (Cornell University)|May 14, 2019

Reinforcement Learning in Robotics39 citations

TL;DR

The paper introduces CORE-RL, a functional regularization approach that mixes a deep RL policy with a control prior to reduce variance and preserve stability during learning in continuous control tasks.

ABSTRACT

Dealing with high variance is a significant challenge in model-free reinforcement learning (RL). Existing methods are unreliable, exhibiting high variance in performance from run to run using different initializations/seeds. Focusing on problems arising in continuous control, we propose a functional regularization approach to augmenting model-free RL. In particular, we regularize the behavior of the deep policy to be similar to a policy prior, i.e., we regularize in function space. We show that functional regularization yields a bias-variance trade-off, and propose an adaptive tuning strategy to optimize this trade-off. When the policy prior has control-theoretic stability guarantees, we further show that this regularization approximately preserves those stability guarantees throughout learning. We validate our approach empirically on a range of settings, and demonstrate significantly reduced variance, guaranteed dynamic stability, and more efficient learning than deep RL alone.

Motivation & Objective

• Motivate reducing high variance in model-free reinforcement learning for continuous control.
• Propose a policy regularization framework that blends RL with a control prior in function space.
• Show that the regularization yields a bias-variance trade-off and provides stability guarantees with robust priors.
• Develop an adaptive strategy to tune the regularization weight based on learning signals.
• Empirically validate variance reduction, stability, and learning efficiency across multiple benchmarks.

Proposed method

• Define a mixed policy that blends the RL policy with a control prior via u_k(s) = (1/(1+λ)) u_{θ_k}(s) + (λ/(1+λ)) u_prior(s).
• Show that this mixed policy is equivalent to a functional regularizer on the RL policy with weight λ.
• Prove a bias-variance trade-off: variance reduces by a factor (1/(1+λ))^2, while bias depends on prior sub-optimality (D_sub).
• Provide an adaptive scheme for λ based on TD-error magnitude to reflect learning confidence (λ(s) = λ_max(1 - e^{-C|δ|})).
• Establish control-theoretic stability guarantees using H-infinity priors and Lyapunov analysis to bound the stable region S_st during learning.
• Describe the CORE-RL algorithm steps: compute control prior, determine λ adaptively or fixed, deploy mixed policy, collect data, update RL policy.

Experimental results

Research questions

• RQ1 Can a control-based functional regularizer reduce variance in policy gradient RL without sacrificing too much performance?
• RQ2 Do stability guarantees from a robust control prior hold throughout learning when using CORE-RL?
• RQ3 Does an adaptive λ, driven by TD error, improve learning efficiency and robustness across tasks?
• RQ4 How does CORE-RL perform compared to baseline RL and the control prior on real data and simulation benchmarks?
• RQ5 What is the impact of regularization on the explorable state region and final policy bias?

Key findings

• CORE-RL significantly reduces variance in policy updates compared to baseline deep RL.
• The approach maintains control-theoretic stability guarantees throughout learning when using a robust control prior.
• Adaptive λ based on TD error improves learning efficiency and final performance across tasks.
• Empirical results on cartpole, car-following with experimental data, and TORCS show improved rewards over the control prior and RL baselines.
• Intermediate regularization weights yield best learning performance with lower variance; too high λ over-constrains exploration.
• Theoretical analysis links regularization strength to the explorable state region and bias-variance trade-off.