[Paper Review] Episodic Linear Quadratic Regulators with Low-rank Transitions

Tianyu Wang, Lin F. Yang|arXiv (Cornell University)|Nov 3, 2020

Advanced Bandit Algorithms Research42 references4 citations

TL;DR

This paper proposes a model-based reinforcement learning algorithm for episodic Linear Quadratic Regulators (LQR) with low-rank system dynamics. By combining optimistic control with low-rank approximation via PCA, the algorithm achieves a regret bound of $\widetilde{\mathcal{O}}(m^{3/2}\sqrt{K})$, where $m$ is the intrinsic rank of the system, independent of the ambient state dimension $d$, enabling efficient learning in high-dimensional settings like image-based control.

ABSTRACT

Linear Quadratic Regulators (LQR) achieve enormous successful real-world applications. Very recently, people have been focusing on efficient learning algorithms for LQRs when their dynamics are unknown. Existing results effectively learn to control the unknown system using number of episodes depending polynomially on the system parameters, including the ambient dimension of the states. These traditional approaches, however, become inefficient in common scenarios, e.g., when the states are high-resolution images. In this paper, we propose an algorithm that utilizes the intrinsic system low-rank structure for efficient learning. For problems of rank-$m$, our algorithm achieves a $K$-episode regret bound of order $\widetilde{O}(m^{3/2} K^{1/2})$. Consequently, the sample complexity of our algorithm only depends on the rank, $m$, rather than the ambient dimension, $d$, which can be orders-of-magnitude larger.

Motivation & Objective

• To address the inefficiency of existing LQR learning algorithms that scale polynomially with ambient state dimension $d$, especially in high-dimensional settings like image-based control.
• To investigate whether sample complexity can be reduced by exploiting low-rank structure in system dynamics.
• To design an online learning algorithm that achieves sublinear regret depending only on the intrinsic rank $m$ of the system, rather than $d$.
• To enable efficient learning in episodic LQR problems where states are high-dimensional but have low intrinsic dimensionality.

Proposed method

• The algorithm uses a model-based approach, estimating system dynamics $M = [A\; B]$ via least-squares regression on collected trajectories.
• It applies principal component analysis (PCA) to project state-action data into a low-dimensional subspace of rank $m$, capturing the intrinsic dynamics.
• An optimistic control policy is computed using uncertainty estimates over the low-rank model, balancing exploration and exploitation via the OFU (Optimism in the Face of Uncertainty) principle.
• The algorithm projects all data points to the PCA subspace from the final episode to stabilize subspace estimation across episodes.
• It employs a self-normalized random process analysis to bound the cumulative uncertainty, ensuring regret independence from ambient dimension $d$.
• The final regret bound is derived using matrix determinant inequalities and logarithmic trace bounds, leveraging the low-rank structure to decouple dependence on $d$.

Experimental results

Research questions

• RQ1 Can we design an LQR learning algorithm whose sample complexity depends only on the intrinsic rank $m$ of the system, rather than the ambient dimension $d$?
• RQ2 How can we effectively combine low-rank approximation with online control learning to maintain regret guarantees?
• RQ3 What is the fundamental regret bound achievable when the system dynamics are low-rank but the observation space is high-dimensional?
• RQ4 Can we maintain sublinear regret in episodic LQR when the underlying state space has low intrinsic dimensionality, even if the observed states are high-dimensional?
• RQ5 How can we handle the instability introduced by changing PCA subspaces across episodes in a sequential learning setting?

Key findings

• The proposed algorithm achieves a $K$-episode regret bound of $\widetilde{\mathcal{O}}(m^{3/2}\sqrt{K})$, which depends only on the intrinsic rank $m$ of the system dynamics.
• The regret bound is independent of the ambient state dimension $d$, making it suitable for high-dimensional inputs such as images.
• The algorithm's sample complexity is polynomial in $m$, significantly improving over prior methods that scale with $\mathrm{poly}(d)$.
• The analysis establishes a novel bound on the determinant of a rank-deficient self-normalized process, enabling the $m$-dependence in regret.
• Empirical results on image-based control tasks (e.g., Cart-Pole) confirm that the algorithm generalizes well despite high-dimensional observations.
• The method outperforms standard LQR learning baselines in sample efficiency when the underlying system has low intrinsic dimensionality.