[Paper Review] Deep Reinforcement Learning Discovers Internal Models

Nir Baram, Tom Zahavy+1 more|arXiv (Cornell University)|Jun 16, 2016

Reinforcement Learning in Robotics16 references3 citations

TL;DR

This paper introduces the Semi-Aggregated Markov Decision Process (SAMDP), a model that automatically extracts hierarchical spatio-temporal structures—such as skills and sub-goals—from deep reinforcement learning policies using temporal-aware clustering on t-SNE visualizations of DQN activations. The method enables interpretable policy analysis and boosts performance via a shared autonomy mechanism, achieving up to 36% higher scores in Atari games by triggering expert intervention when policy failure is predicted.

ABSTRACT

Deep Reinforcement Learning (DRL) is a trending field of research, showing great promise in challenging problems such as playing Atari, solving Go and controlling robots. While DRL agents perform well in practice we are still lacking the tools to analayze their performance. In this work we present the Semi-Aggregated MDP (SAMDP) model. A model best suited to describe policies exhibiting both spatial and temporal hierarchies. We describe its advantages for analyzing trained policies over other modeling approaches, and show that under the right state representation, like that of DQN agents, SAMDP can help to identify skills. We detail the automatic process of creating it from recorded trajectories, up to presenting it on t-SNE maps. We explain how to evaluate its fitness and show surprising results indicating high compatibility with the policy at hand. We conclude by showing how using the SAMDP model, an extra performance gain can be squeezed from the agent.

Motivation & Objective

• To automatically discover internal hierarchical structures—such as skills and sub-goals—within trained DRL policies without manual feature engineering.
• To develop a model that provides human-interpretable explanations of DRL agents' behavior by identifying spatial and temporal abstractions in state representations.
• To enable performance improvement through a shared autonomy framework that detects and intervenes in high-risk policy behaviors using the SAMDP model.
• To evaluate the fidelity of the extracted model using quantitative criteria such as value function consistency and policy-reward correlation.

Proposed method

• Construct t-SNE maps from DQN neural activation trajectories to visualize state representations in a lower-dimensional space.
• Apply temporal-aware clustering algorithms to group states into semi-aggregated states that represent distinct skills or sub-goals.
• Estimate transition probabilities and reward functions empirically from trajectories to build the SAMDP model.
• Use three evaluation criteria: value function consistency (VMSE), correlation between greedy policy actions and trajectory rewards, and correlation of policy transitions with high- vs. low-reward trajectories.
• Implement a shared autonomy mechanism: if the likelihood of a trajectory matching low-reward behavior (T⁻) exceeds that of high-reward behavior (T⁺), trigger an 'eject' button to terminate and intervene.
• Use the SAMDP model to predict failure-prone states and guide expert intervention without retraining the DQN.

Experimental results

Research questions

• RQ1 Can a data-driven, automatic method identify meaningful skills and sub-goals in DQN policies using only state activation trajectories?
• RQ2 How well does the SAMDP model capture the true dynamics and reward structure of the underlying MDP?
• RQ3 Can the SAMDP model be used to detect when a DQN policy is likely to fail, enabling timely expert intervention?
• RQ4 To what extent can the SAMDP model improve policy performance without retraining the original DQN agent?

Key findings

• The SAMDP model achieved a 36% average performance improvement in Breakout by triggering expert intervention when the policy was predicted to fail.
• The model showed strong value function consistency, with high correlation (VMSE) between DQN Q-values and SAMDP-predicted values across states.
• A positive correlation (avg. corr_i > 0) was observed between greedy policy actions and high-reward trajectories, confirming model consistency.
• The greedy policy's transition matrix correlated more strongly with high-reward trajectories (T⁺) than with low-reward ones (T⁻), validating model fidelity.
• The method successfully identified well-defined skills with clear initiation and termination sets, even without manual labeling or supervision.
• The model enabled performance gains without retraining, demonstrating the feasibility of shared autonomy using internal model discovery.