[Paper Review] Approximate information state for approximate planning and reinforcement learning in partially observed systems

Jayakumar Subramanian, Amit Kumar Sinha+2 more|arXiv (Cornell University)|Oct 17, 2020

Receptor Mechanisms and Signaling31 citations

TL;DR

The paper develops a rigorous framework of approximate information states (AIS) for planning and reinforcement learning in partially observed systems, derives performance bounds, shows AIS encompasses many known approximations, and introduces AIS-based policy gradient methods with multi-time scale learning, validated by experiments.

ABSTRACT

We propose a theoretical framework for approximate planning and learning in partially observed systems. Our framework is based on the fundamental notion of information state. We provide two equivalent definitions of information state -- i) a function of history which is sufficient to compute the expected reward and predict its next value; ii) equivalently, a function of the history which can be recursively updated and is sufficient to compute the expected reward and predict the next observation. An information state always leads to a dynamic programming decomposition. Our key result is to show that if a function of the history (called approximate information state (AIS)) approximately satisfies the properties of the information state, then there is a corresponding approximate dynamic program. We show that the policy computed using this is approximately optimal with bounded loss of optimality. We show that several approximations in state, observation and action spaces in literature can be viewed as instances of AIS. In some of these cases, we obtain tighter bounds. A salient feature of AIS is that it can be learnt from data. We present AIS based multi-time scale policy gradient algorithms. and detailed numerical experiments with low, moderate and high dimensional environments.

Motivation & Objective

• Formalize information state as a statistic of history sufficient for performance evaluation and dynamic programming.
• Introduce approximate information state (AIS) as a compressed history that approximately preserves these properties.
• Derive bounds showing AIS-based planning incurs bounded loss compared to optimal policies.
• Show AIS encompasses existing approximations in state, observation, and action spaces and can be learned from data.
• Extend AIS to decentralized multi-agent settings and develop AIS-based online policy gradient algorithms.

Proposed method

• Define information state via two equivalent conditions: (P1) sufficiency for performance evaluation, and (P2) sufficiency to predict future statistics; provide equivalent dynamic programming decompositions.
• Propose AIS as a compression of history that approximately satisfies (P1) and (P2); derive bounds on performance loss under AIS.
• Present two equivalent AIS formulations corresponding to the two information state definitions, enabling two parallel DP constructions.
• Extend the framework to approximate actions spaces and to decentralized (multi-agent) systems using common-information based AIS.
• Develop AIS-based policy gradient algorithms that learn AIS representations via multi-timescale stochastic optimization.
• Provide theoretical results showing AIS-based policies are near-optimal when the AIS closely satisfies the information-state properties.

Experimental results

Research questions

• RQ1 What is an information state, and how can it enable dynamic programming for partially observed systems?
• RQ2 How can we define and quantify an approximate information state (AIS) that compresses history while preserving near-optimal planning performance?
• RQ3 What are the bounds on performance loss when using AIS instead of the exact information state?
• RQ4 Can existing approximations in state, observation, and action spaces be viewed as AIS instances, and do they yield tighter guarantees?
• RQ5 How can AIS be learned from data and used to design online policy gradient algorithms for PORL, including multi-agent settings?

Key findings

• An AIS provides a principled compression of history that yields an approximate dynamic program with bounded loss of optimality.
• Many existing approximation schemes in state, observation, and action spaces are special cases of AIS, with some cases offering tighter bounds.
• AIS can be learned from data, enabling data-driven PORL with theoretical guarantees on performance degradation.
• The framework extends to decentralized systems using common-information AIS, enabling multi-agent PORL analyses.
• AIS-based online policy gradient methods with multiple learning timescales are derived and validated on a range of environments.