[Paper Review] Robust exploration in linear quadratic reinforcement learning

Jack Umenberger, Mina Ferizbegovic+2 more|arXiv (Cornell University)|Jun 4, 2019

Reinforcement Learning in Robotics36 references4 citations

TL;DR

This paper proposes a convex optimization-based method for robust reinforcement learning in linear quadratic control problems, balancing exploitation and targeted exploration to minimize worst-case costs under model uncertainty. The approach dynamically excites the system to reduce uncertainty in the most critical parameters, achieving superior robustness and performance over baseline methods in both simulations and hardware-in-the-loop experiments.

ABSTRACT

This paper concerns the problem of learning control policies for an unknown linear dynamical system to minimize a quadratic cost function. We present a method, based on convex optimization, that accomplishes this task robustly: i.e., we minimize the worst-case cost, accounting for system uncertainty given the observed data. The method balances exploitation and exploration, exciting the system in such a way so as to reduce uncertainty in the model parameters to which the worst-case cost is most sensitive. Numerical simulations and application to a hardware-in-the-loop servo-mechanism demonstrate the approach, with appreciable performance and robustness gains over alternative methods observed in both.

Motivation & Objective

• To address the exploration-exploitation trade-off in learning control policies for unknown linear dynamical systems with quadratic costs.
• To ensure robust performance by minimizing the worst-case cost under model uncertainty derived from observed data.
• To design exploration strategies that are targeted, reducing uncertainty only in parameters that most affect worst-case cost.
• To develop a computationally tractable, convex approximation for robust LQR that enables efficient policy synthesis.
• To validate the method in both numerical simulations and hardware-in-the-loop experiments, demonstrating improved performance and robustness.

Proposed method

• Derives a high-probability bound on the spectral norm of the system parameter estimation error, enabling uncertainty quantification for robust control.
• Proposes a convex semidefinite program (SDP) approximation to the worst-case infinite-horizon LQR problem under parameter uncertainty.
• Integrates the uncertainty bound and convex LQR approximation into a robust reinforcement learning (RRL) framework that balances exploration and exploitation.
• Uses a look-ahead horizon of h=10 (simulations) or h=5 (hardware) to guide policy updates based on future worst-case cost sensitivity.
• Optimizes exploration variance adaptively to reduce uncertainty in the most critical model parameters, avoiding arbitrary or random probing.
• Employs a nominal robust policy as a baseline and compares against a greedy exploration policy that increases variance to match theoretical worst-case cost.

Experimental results

Research questions

• RQ1 How can exploration be systematically targeted to reduce uncertainty in parameters that most impact worst-case control performance?
• RQ2 Can a convex optimization framework be used to approximate the worst-case LQR problem under model uncertainty in a tractable way?
• RQ3 Does robust reinforcement learning that combines targeted exploration with worst-case cost minimization outperform purely exploitative or randomly exploratory policies?
• RQ4 To what extent does the proposed method improve robustness and performance in real-world hardware-in-the-loop control scenarios?
• RQ5 How does the information gain (uncertainty reduction) from the proposed method compare to greedy exploration, despite lower overall uncertainty reduction?

Key findings

• The proposed RRL method achieves lower total cost than both the nominal (pure exploitation) and greedy (random exploration) policies in numerical simulations.
• In the initial epochs, the nominal policy performs better due to lack of exploration, but RRL surpasses it in later stages, demonstrating effective long-term learning.
• RRL reduces uncertainty more strategically than greedy exploration, achieving higher information gain than nominal but lower than greedy, yet still outperforms both in cost.
• In hardware-in-the-loop experiments, RRL significantly outperforms both baseline methods in terms of total cost and cost per epoch.
• The method achieves robust performance even when model uncertainty is updated using theoretical worst-case behavior, confirming its worst-case optimization goal.
• The information metric (1/λ_max(D_i^{-1})) shows that RRL reduces uncertainty in a way that directly correlates with cost reduction, indicating targeted learning.