[Paper Review] Information Theoretic Regret Bounds for Online Nonlinear Control.

Sham M. Kakade, Akshay Krishnamurthy+3 more|arXiv (Cornell University)|Oct 1, 2020

Advanced Bandit Algorithms Research13 citations

TL;DR

This paper proposes the Lower Confidence-based Continuous Control (LC3) algorithm for online nonlinear control in unknown dynamical systems modeled as functions in a Reproducing Kernel Hilbert Space (RKHS). It achieves a near-optimal O(√T) regret bound independent of system dimension, relying on information-theoretic quantities, and demonstrates improved learning through effective exploration in nonlinear control tasks.

ABSTRACT

This work studies the problem of sequential control in an unknown, nonlinear dynamical system, where we model the underlying system dynamics as an unknown function in a known Reproducing Kernel Hilbert Space. This framework yields a general setting that permits discrete and continuous control inputs as well as non-smooth, non-differentiable dynamics. Our main result, the Lower Confidence-based Continuous Control (LC3) algorithm, enjoys a near-optimal O(\sqrt{T}) regret bound against the optimal controller in episodic settings, where T is the number of episodes. The bound has no explicit dependence on dimension of the system dynamics, which could be infinite, but instead only depends on information theoretic quantities. We empirically show its application to a number of nonlinear control tasks and demonstrate the benefit of exploration for learning model dynamics.

Motivation & Objective

• To address online control in unknown, nonlinear dynamical systems where dynamics are modeled as functions in a known Reproducing Kernel Hilbert Space (RKHS).
• To develop a control algorithm that achieves near-optimal regret in episodic settings despite non-smooth, non-differentiable dynamics.
• To eliminate explicit dependence on system dimension in regret bounds by relying on information-theoretic quantities instead.
• To empirically validate the effectiveness of exploration in learning complex, nonlinear system dynamics.

Proposed method

• Model the unknown system dynamics as an unknown function in a known Reproducing Kernel Hilbert Space (RKHS), enabling flexible representation of nonlinear and non-smooth dynamics.
• Design the LC3 algorithm using lower confidence bounds to balance exploration and exploitation in sequential control decisions.
• Derive regret bounds based on information-theoretic quantities such as mutual information between observations and system dynamics.
• Ensure the regret bound scales as O(√T) with no explicit dependence on the dimension of the dynamics, even if infinite.
• Integrate exploration strategies that improve model learning by actively probing uncertain regions of the dynamics.
• Apply the algorithm to various nonlinear control tasks to empirically demonstrate its effectiveness.

Experimental results

Research questions

• RQ1 Can we achieve near-optimal regret in online nonlinear control without explicit dependence on system dimension?
• RQ2 How can information-theoretic quantities be leveraged to bound regret in unknown, nonlinear dynamical systems?
• RQ3 What role does structured exploration play in learning accurate dynamics in non-smooth, non-differentiable systems?
• RQ4 Can a kernel-based function space model effectively represent complex nonlinear dynamics while enabling efficient control?

Key findings

• The LC3 algorithm achieves a regret bound of O(√T), which is near-optimal for episodic online control settings.
• The regret bound does not explicitly depend on the dimension of the system dynamics, even allowing for infinite-dimensional RKHS.
• The bound is derived using information-theoretic quantities such as mutual information, reflecting the uncertainty in system dynamics.
• Empirical results show that exploration strategies in LC3 significantly improve learning of nonlinear system dynamics.
• The framework successfully handles both discrete and continuous control inputs, as well as non-smooth and non-differentiable dynamics.