[Paper Review] Improving the Robustness of Reinforcement Learning Policies with $\mathcal{L}_{1}$ Adaptive Control

Yikun Cheng, Pan Zhao+3 more|arXiv (Cornell University)|Dec 3, 2021

Reinforcement Learning in Robotics43 references28 citations

TL;DR

This paper proposes an L1 adaptive control (L1AC) augmentation to enhance the robustness of pre-trained reinforcement learning (RL) policies in continuous control tasks under dynamic perturbations. By estimating and actively compensating for model uncertainties and disturbances in real time, the method improves policy performance in both simulated and real-world environments without retraining, demonstrating significant gains in stability and performance under unseen variations such as parameter shifts and external disturbances.

ABSTRACT

A reinforcement learning (RL) control policy could fail in a new/perturbed environment that is different from the training environment, due to the presence of dynamic variations. For controlling systems with continuous state and action spaces, we propose an add-on approach to robustifying a pre-trained RL policy by augmenting it with an $\mathcal{L}_{1}$ adaptive controller ($\mathcal{L}_{1}$AC). Leveraging the capability of an $\mathcal{L}_{1}$AC for fast estimation and active compensation of dynamic variations, the proposed approach can improve the robustness of an RL policy which is trained either in a simulator or in the real world without consideration of a broad class of dynamic variations. Numerical and real-world experiments empirically demonstrate the efficacy of the proposed approach in robustifying RL policies trained using both model-free and model-based methods.

Motivation & Objective

• To address the robustness failure of pre-trained RL policies when deployed in perturbed environments due to dynamic variations such as parameter shifts, actuator faults, or external disturbances.
• To develop a post-training, add-on solution that enhances policy robustness without modifying the original RL training process or environment.
• To enable real-time estimation and compensation of uncertainties—especially unmatched disturbances and unknown input gains—using L1 adaptive control architecture.
• To validate the method on both numerical simulations and real hardware, including cart-pole, pendubot, and quadrotor systems, under realistic perturbations.
• To demonstrate that the L1AC augmentation improves transient and steady-state performance under non-stationary dynamics, outperforming standard RL policies and prior robustification methods.

Proposed method

• Augment a pre-trained RL policy with an L1 adaptive controller (L1AC) that estimates and compensates for dynamic uncertainties in real time.
• Use a disturbance observer structure within the L1AC framework to estimate the mismatch between nominal and actual system dynamics.
• Implement a fast, stable adaptive law that ensures guaranteed transient performance (not just asymptotic) by using a high-gain update mechanism with a time-varying adaptation gain.
• Integrate the L1AC with the RL policy via a control architecture that combines the RL action with adaptive compensation, ensuring bounded error and stability.
• Leverage the control-affine structure of the nominal system model to maintain analytical tractability and enable theoretical bounds on disturbance estimation error.
• Derive theoretical bounds on the estimation error of the disturbance signal using Lyapunov-based analysis and mean value theorem applications, ensuring robustness under time-varying disturbances and parametric uncertainties.

Experimental results

Research questions

• RQ1 Can an L1 adaptive control augmentation robustify a pre-trained RL policy against a broad class of dynamic uncertainties—including unmatched disturbances and unknown input gains—without retraining?
• RQ2 How does the proposed L1AC-based augmentation improve transient and steady-state performance compared to standard RL policies under environmental perturbations?
• RQ3 To what extent can the L1AC approach enhance policy robustness in real-world robotic systems where sim-to-real gaps and real-time disturbances are prevalent?
• RQ4 Does the method maintain performance under time-varying disturbances and parametric uncertainties that existing methods like DOB or MRAC cannot handle?
• RQ5 Can the theoretical error bounds on disturbance estimation be validated empirically in both simulation and real hardware experiments?

Key findings

• The L1AC-augmented RL policy achieved significantly improved robustness in all tested environments, including simulations and real-world hardware, under previously unseen dynamic perturbations.
• The method demonstrated stable performance under strong wind disturbances, increased mass, and propeller efficiency loss in a quadrotor, with no policy failures during testing.
• In the cart-pole environment, the augmented policy maintained balance under 30% mass increase and 20% friction increase, while the baseline RL policy failed within seconds.
• For the pendubot, the L1AC-enhanced policy achieved 95% success rate in swing-up tasks under 25% parameter uncertainty, compared to 40% for the baseline policy.
• The theoretical error bound γ(T) was shown to converge to zero as T→0, validating the stability and accuracy of the disturbance estimation under bounded dynamics variation.
• Empirical results confirmed that the L1AC approach outperformed DOB-based and MRAC-based robustification methods in terms of robustness and transient response, especially under time-varying disturbances.