[Paper Review] Efficient Reinforcement Learning for High Dimensional Linear Quadratic Systems

Morteza Ibrahimi, Adel Javanmard+1 more|arXiv (Cornell University)|Mar 24, 2013

Advanced Bandit Algorithms Research13 references17 citations

TL;DR

This paper proposes an adaptive reinforcement learning algorithm for high-dimensional linear quadratic (LQ) systems with sparse dynamics, achieving a regret bound of $O(p\sqrt{T})$—significantly improving over prior $O(p^{p+r+2}\sqrt{T})$ bounds. By leveraging sparsity in system matrices and using optimistic parameter estimation, the algorithm attains $ (1+\epsilon) $-optimal average cost in polylogarithmic time in $ p $, enabling efficient control in high-dimensional settings like online advertising.

ABSTRACT

We study the problem of adaptive control of a high dimensional linear quadratic (LQ) system. Previous work established the asymptotic convergence to an optimal controller for various adaptive control schemes. More recently, for the average cost LQ problem, a regret bound of ${O}(\sqrt{T})$ was shown, apart form logarithmic factors. However, this bound scales exponentially with $p$, the dimension of the state space. In this work we consider the case where the matrices describing the dynamic of the LQ system are sparse and their dimensions are large. We present an adaptive control scheme that achieves a regret bound of ${O}(p \sqrt{T})$, apart from logarithmic factors. In particular, our algorithm has an average cost of $(1+\eps)$ times the optimum cost after $T = \polylog(p) O(1/\eps^2)$. This is in comparison to previous work on the dense dynamics where the algorithm requires time that scales exponentially with dimension in order to achieve regret of $\eps$ times the optimal cost. We believe that our result has prominent applications in the emerging area of computational advertising, in particular targeted online advertising and advertising in social networks.

Motivation & Objective

• To address the poor scaling of existing reinforcement learning algorithms for high-dimensional linear quadratic (LQ) systems, where regret grows exponentially with state dimension $ p $.
• To develop an adaptive control scheme that achieves near-optimal performance with minimal regret in high-dimensional settings where $ p \gg 1 $.
• To exploit sparsity in the system matrices $ A^0 $ and $ B^0 $ to enable accurate and efficient parameter estimation with only $ O(\log(p+r)) $ observations.
• To establish a regret bound that scales gracefully with dimension $ p $, achieving $ (1+\epsilon) $-optimal cost in $ \text{polylog}(p) \cdot O(1/\epsilon^2) $ time.

Proposed method

• Uses the optimism in the face of uncertainty (OFU) principle to select the most optimistic parameter estimate within a confidence set, ensuring exploration and convergence to optimal control.
• Constructs a confidence set $ \Omega_t $ around estimated system parameters $ \widetilde{\Theta}_t $, ensuring $ \Theta^0 \in \Omega_t $ with high probability.
• Employs a cost decomposition technique to relate the cumulative cost to the optimal cost via three error terms: $ C_1, C_2, C_3 $, which are bounded under high-probability events.
• Introduces two key events: $ \mathcal{E}_1 $ (true parameters in confidence set) and $ \mathcal{E}_2 $ (bounded noise), to derive high-probability bounds on regret.
• Applies concentration inequalities and spectral norm bounds on the Riccati operator $ K(\widetilde{\Theta}_t) $ to control the growth of estimation and control errors.
• Leverages sparsity in $ A^0 $ and $ B^0 $ to reduce the effective number of parameters, enabling accurate estimation with few samples and improving regret scaling.

Experimental results

Research questions

• RQ1 Can reinforcement learning algorithms achieve regret bounds that scale polynomially with dimension $ p $ in high-dimensional LQ systems?
• RQ2 Does exploiting sparsity in system dynamics significantly improve the sample efficiency and regret scaling of adaptive control algorithms?
• RQ3 Can an OFU-based algorithm achieve $ (1+\epsilon) $-optimal performance in time that is polylogarithmic in $ p $, rather than exponential?
• RQ4 How does the regret scale with dimension when the system matrices are sparse, compared to the dense case?

Key findings

• The proposed algorithm achieves a regret bound of $ O(p\sqrt{T}) $, up to logarithmic factors, which is a significant improvement over the prior $ O(p^{p+r+2}\sqrt{T}) $ bound for dense systems.
• The algorithm attains an average cost within $ (1+\epsilon) $ of the optimal cost after $ T = \text{polylog}(p) \cdot O(1/\epsilon^2) $ time steps, enabling fast convergence in high dimensions.
• Under the assumption of sparse system matrices, accurate parameter estimation is possible with only $ O(\log(p+r)) $ observations, enabling efficient learning.
• The regret bound is derived through high-probability decomposition into three error terms $ C_1, C_2, C_3 $, each bounded using concentration and spectral norm techniques.
• The analysis shows that the regret scales linearly with $ p $, not exponentially, due to the exploitation of sparsity and careful confidence set design.