[Paper Review] Provably efficient RL with Rich Observations via Latent State Decoding

Simon S. Du, Akshay Krishnamurthy+4 more|arXiv (Cornell University)|Jan 25, 2019

Machine Learning and Algorithms58 citations

TL;DR

Proposes a provably sample-efficient RL approach for rich-observation MDPs by explicitly learning a decoding from observations to latent states and constructing an exploratory policy cover with finite-sample guarantees. The method leverages backward probability vectors and inductive decoding to reduce to a tractable latent-state exploration problem.

ABSTRACT

We study the exploration problem in episodic MDPs with rich observations generated from a small number of latent states. Under certain identifiability assumptions, we demonstrate how to estimate a mapping from the observations to latent states inductively through a sequence of regression and clustering steps -- where previously decoded latent states provide labels for later regression problems -- and use it to construct good exploration policies. We provide finite-sample guarantees on the quality of the learned state decoding function and exploration policies, and complement our theory with an empirical evaluation on a class of hard exploration problems. Our method exponentially improves over $Q$-learning with naïve exploration, even when $Q$-learning has cheating access to latent states.

Motivation & Objective

• Motivate and address exploration in episodic MDPs with rich observations emitted from a small latent state space.
• Introduce a tractable latent-state decoding approach that enables efficient exploration without dependency on the full observation space.
• Provide finite-sample guarantees on decoding accuracy and the quality of exploration policies.
• Show empirical validation demonstrating strong exploration performance on hard problems beyond naïve baselines.

Proposed method

• Formulate the block Markov decision process (BMDP) capturing latent states, observable contexts, and transitions.
• Embed contexts and latent states into a shared low-dimensional space using g(x) and φ(s) in Δ_MK, under a realizability assumption for a decoding function class.
• Use backward probability vectors bν(s′) to represent latent states and establish γ-separability to distinguish latent states via these vectors.
• Solve a sequence of least-squares problems via an ERM oracle to learn context embeddings and derive decoding functions.
• Cluster embedding vectors to identify latent-state blocks and define a decoding map f̂ to map contexts to discovered latent states.
• Construct an ε-policy cover by estimating transition probabilities and applying dynamic programming to reach target latent states.
• Iterate level-by-level (h = 2,…,H+1) to build latent-state sets, embeddings, transition estimates, and policy sets, ensuring coverage and accuracy bounds.

Experimental results

Research questions

• RQ1 Can rich observations be effectively decoded into a small latent-state space under manageable separability conditions?
• RQ2 What are the finite-sample guarantees for decoding accuracy and the resulting policy cover in a BMDP with rich observations?
• RQ3 How can backward conditional probabilities be leveraged to learn latent-state embeddings via regression?
• RQ4 How does the proposed inductive decoding approach compare to naïve exploration and baseline RL methods in terms of sample efficiency?
• RQ5 What is the role of the γ-separability margin and μ_min (minimum reaching probability) in the sample complexity?

Key findings

• The paper provides finite-sample guarantees for recovering a latent-state decoding function and an ε-policy cover under separability assumptions.
• The PCID algorithm achieves a policy cover whose size is O(MH) with high probability, using a sample complexity that scales polylogarithmically with observation space size and polynomially with M, K, and H.
• The backward probability vector formalism enables the decoding step via least-squares regression, yielding accurate state embeddings that align with latent states.
• In deterministic-BMDPs, the ε parameter can be zero, simplifying decoding and enabling exact state reachability with fixed-action sequences.
• Empirical results show substantial exploration efficiency gains over naïve Q-learning, even when baselines have cheating access to latent states.