[Paper Review] Reinforcement Learning in Rich-Observation MDPs using Spectral Methods

Kamyar Azizzadenesheli, Alessandro Lazaric+1 more|arXiv (Cornell University)|Nov 11, 2016

Reinforcement Learning in Robotics30 references12 citations

TL;DR

This paper proposes SL-UCRL, a reinforcement learning algorithm for Rich-Observation MDPs (ROMDPs) that uses spectral decomposition to learn an injective mapping from high-dimensional observations to a low-dimensional hidden state space. By clustering observations into latent states and applying UCRL on the estimated hidden MDP, the algorithm achieves regret asymptotically matching the oracle UCRL algorithm, with regret growing only with the number of hidden states, not the observation space size.

ABSTRACT

Reinforcement learning (RL) in Markov decision processes (MDPs) with large state spaces is a challenging problem. The performance of standard RL algorithms degrades drastically with the dimensionality of state space. However, in practice, these large MDPs typically incorporate a latent or hidden low-dimensional structure. In this paper, we study the setting of rich-observation Markov decision processes (ROMDP), where there are a small number of hidden states which possess an injective mapping to the observation states. In other words, every observation state is generated through a single hidden state, and this mapping is unknown a priori. We introduce a spectral decomposition method that consistently learns this mapping, and more importantly, achieves it with low regret. The estimated mapping is integrated into an optimistic RL algorithm (UCRL), which operates on the estimated hidden space. We derive finite-time regret bounds for our algorithm with a weak dependence on the dimensionality of the observed space. In fact, our algorithm asymptotically achieves the same average regret as the oracle UCRL algorithm, which has the knowledge of the mapping from hidden to observed spaces. Thus, we derive an efficient spectral RL algorithm for ROMDPs.

Motivation & Objective

• To address the challenge of high regret in standard RL algorithms when state spaces are large and high-dimensional.
• To exploit low-dimensional latent structure in large observation MDPs to improve learning efficiency and regret bounds.
• To develop a method that learns the hidden state mapping from observations without prior knowledge of the injective mapping.
• To integrate spectral clustering into an optimistic RL framework (UCRL) to maintain regret guarantees.
• To achieve regret that scales with the number of hidden states rather than the number of observations, asymptotically matching the oracle case.

Proposed method

• Uses tensor-based spectral decomposition on observation trajectories to estimate the mapping from observations to hidden states.
• Applies the spectral clustering method of Anandkumar et al. (2014) to identify clusters corresponding to hidden states with high probability.
• Constructs an auxiliary MDP on the estimated hidden states and applies UCRL for exploration-exploitation trade-off.
• Employs confidence bounds based on Hoeffding-type inequalities to ensure accurate transition probability estimation in the auxiliary MDP.
• Uses epoch-based learning where the mapping is updated periodically using collected samples, improving clustering accuracy over time.
• Integrates the estimated hidden state space into UCRL to reduce computational complexity from O(Y³) to O(X³), where Y is observation space size and X is hidden state count.

Experimental results

Research questions

• RQ1 Can spectral methods reliably recover the hidden state structure in large observation MDPs with unknown injective mappings?
• RQ2 Does learning in the estimated hidden space lead to regret comparable to that of an oracle algorithm with full knowledge of the hidden structure?
• RQ3 Can the algorithm achieve low regret with only weak dependence on the observation space size Y?
• RQ4 How does the regret scale with the number of hidden states X and the time horizon N?
• RQ5 What is the impact of clustering accuracy on the overall learning performance and regret?

Key findings

• The SL-UCRL algorithm achieves a regret bound that asymptotically matches the oracle UCRL algorithm, which has full knowledge of the hidden-to-observation mapping.
• The regret of SL-UCRL depends only on the number of hidden states X, not the observation space size Y, resulting in significantly improved scalability.
• The time and space complexity of the algorithm decrease from O(Y³) to O(X³) as clustering improves, due to the reduced size of the auxiliary MDP.
• The number of steps required to learn the correct clustering is bounded and does not grow with the time horizon N, leading to a constant regret during the clustering phase.
• Empirical results show SL-UCRL outperforms UCRL and DQN in terms of regret and robustness, even in random MDPs without true hidden structure.
• The algorithm successfully identifies approximate clusters in practice, even when the true hidden structure is absent, due to the spectral method’s ability to detect latent groupings.