[논문 리뷰] Control Regularization for Reduced Variance Reinforcement Learning

Richard Cheng, Abhinav Verma외 4명|arXiv (Cornell University)|2019. 05. 14.

Reinforcement Learning in Robotics인용 수 39

한 줄 요약

본 논문은 CORE-RL을 제시한다. 이는 딥 RL 정책과 제어 사전 정보를 혼합하는 기능적 정규화 접근법으로, 연속 제어 과제에서 학습 중 분산을 줄이고 안정성을 보존한다.

ABSTRACT

Dealing with high variance is a significant challenge in model-free reinforcement learning (RL). Existing methods are unreliable, exhibiting high variance in performance from run to run using different initializations/seeds. Focusing on problems arising in continuous control, we propose a functional regularization approach to augmenting model-free RL. In particular, we regularize the behavior of the deep policy to be similar to a policy prior, i.e., we regularize in function space. We show that functional regularization yields a bias-variance trade-off, and propose an adaptive tuning strategy to optimize this trade-off. When the policy prior has control-theoretic stability guarantees, we further show that this regularization approximately preserves those stability guarantees throughout learning. We validate our approach empirically on a range of settings, and demonstrate significantly reduced variance, guaranteed dynamic stability, and more efficient learning than deep RL alone.

연구 동기 및 목표

• 연속 제어를 위한 모델-프리 강화학습에서 높은 분산 감소의 필요성을 동기 부여한다.
• 함수 공간에서 RL과 제어 사전 정보를 혼합하는 정책 정규화 프레임워크를 제안한다.
• 해당 정규화가 바이어스-분산 트레이드오프를 형성하고 강건한 사전 정보를 갖춘 경우 안정성 보장을 제공함을 보인다.
• 학습 신호를 기반으로 정규화 가중치를 조정하는 적응 전략을 개발한다.
• 다수 벤치마크에서 분산 감소, 안정성 및 학습 효율성을 실험적으로 검증한다.

제안 방법

• Define a mixed policy that blends the RL policy with a control prior via u_k(s) = (1/(1+λ)) u_{θ_k}(s) + (λ/(1+λ)) u_prior(s).
• Show that this mixed policy is equivalent to a functional regularizer on the RL policy with weight λ.
• Prove a bias-variance trade-off: variance reduces by a factor (1/(1+λ))^2, while bias depends on prior sub-optimality (D_sub).
• Provide an adaptive scheme for λ based on TD-error magnitude to reflect learning confidence (λ(s) = λ_max(1 - e^{-C|δ|})).
• Establish control-theoretic stability guarantees using H-infinity priors and Lyapunov analysis to bound the stable region S_st during learning.
• Describe the CORE-RL algorithm steps: compute control prior, determine λ adaptively or fixed, deploy mixed policy, collect data, update RL policy.

실험 결과

연구 질문

• RQ1 Can a control-based functional regularizer reduce variance in policy gradient RL without sacrificing too much performance?
• RQ2 Do stability guarantees from a robust control prior hold throughout learning when using CORE-RL?
• RQ3 Does an adaptive λ, driven by TD error, improve learning efficiency and robustness across tasks?
• RQ4 How does CORE-RL perform compared to baseline RL and the control prior on real data and simulation benchmarks?
• RQ5 What is the impact of regularization on the explorable state region and final policy bias?

주요 결과

• CORE-RL significantly reduces variance in policy updates compared to baseline deep RL.
• The approach maintains control-theoretic stability guarantees throughout learning when using a robust control prior.
• Adaptive λ based on TD error improves learning efficiency and final performance across tasks.
• Empirical results on cartpole, car-following with experimental data, and TORCS show improved rewards over the control prior and RL baselines.
• Intermediate regularization weights yield best learning performance with lower variance; too high λ over-constrains exploration.
• Theoretical analysis links regularization strength to the explorable state region and bias-variance trade-off.