[Paper Review] Graying the black box: Understanding DQNs

Tom Zahavy, Nir Ben Zrihem+1 more|arXiv (Cornell University)|Feb 8, 2016

Reinforcement Learning in Robotics38 references58 citations

TL;DR

This paper introduces a methodology to interpret Deep Q-Networks (DQNs) by identifying hierarchical spatio-temporal abstractions through a novel Semi Aggregated Markov Decision Process (SAMDP) model. By learning SAMDP automatically from data, the authors reveal that DQNs implicitly learn state aggregation and options, explaining their success and enabling policy interpretation, debugging, and robustification via an 'eject' mechanism that improves performance by 36%, 20%, and 4.7% in Breakout, Seaquest, and Pacman, respectively.

ABSTRACT

In recent years there is a growing interest in using deep representations for reinforcement learning. In this paper, we present a methodology and tools to analyze Deep Q-networks (DQNs) in a non-blind matter. Moreover, we propose a new model, the Semi Aggregated Markov Decision Process (SAMDP), and an algorithm that learns it automatically. The SAMDP model allows us to identify spatio-temporal abstractions directly from features and may be used as a sub-goal detector in future work. Using our tools we reveal that the features learned by DQNs aggregate the state space in a hierarchical fashion, explaining its success. Moreover, we are able to understand and describe the policies learned by DQNs for three different Atari2600 games and suggest ways to interpret, debug and optimize deep neural networks in reinforcement learning.

Motivation & Objective

• To address the interpretability gap in Deep Q-Networks (DQNs), which are often treated as black boxes despite their success in Atari games.
• To understand how DQNs implicitly learn hierarchical state abstractions and options without explicit engineering.
• To develop tools for debugging and improving DQN policies by analyzing learned representations and dynamics.
• To propose a method for robustifying DQN policies using the SAMDP model to detect and intervene in low-performing behaviors.
• To enable better design and optimization of deep reinforcement learning agents through interpretable, data-driven abstractions.

Proposed method

• Propose the Semi Aggregated Markov Decision Process (SAMDP), an approximation of the true MDP that captures state dynamics and temporal abstractions.
• Learn the SAMDP model automatically from DQN experience replay data using clustering of state representations and transition dynamics.
• Use k-means clustering on DQN-learned features to identify state clusters, then infer transition matrices and reward structures per cluster.
• Evaluate the SAMDP model using metrics like Vector Mean Squared Error (VMSE) and correlation between greedy policies and high/low-reward trajectories.
• Implement an 'eject' mechanism that triggers intervention when test trajectories are more likely to originate from low-reward (bottom-k) trajectories than high-reward ones.
• Apply the SAMDP model to detect policy deterioration and return control to a human or superior agent in critical states, improving overall performance without retraining.

Experimental results

Research questions

• RQ1 How do DQNs implicitly learn hierarchical state abstractions and options without explicit supervision or engineering?
• RQ2 Can we automatically discover a structured, interpretable model of the environment from DQN representations to explain policy behavior?
• RQ3 To what extent can the learned SAMDP model be used to interpret, debug, and improve DQN policies?
• RQ4 Can the SAMDP model detect when a DQN policy is likely to fail, enabling intervention to improve robustness?
• RQ5 How does the performance of a DQN policy improve when combined with an automated detection of low-performing behavior using the SAMDP model?

Key findings

• DQNs learn hierarchical state abstractions by mapping the state space into distinct sub-manifolds where different features dominate, enabling localized policy learning.
• The SAMDP model successfully captures temporal abstractions and options with defined initial and termination conditions, explaining the success of DQN in complex environments.
• The correlation between the greedy policy and top-rewarded trajectories is significantly higher than with bottom-rewarded trajectories, validating the model’s ability to distinguish high-quality behavior.
• The 'eject' mechanism, triggered when behavior aligns more with low-reward trajectories, improved performance by 36% in Breakout, 20% in Seaquest, and 4.7% in Pacman without retraining.
• The SAMDP model enables interpretation of DQN policies through interpretable logic rules derived from neural activations, enhancing debugging and design insights.
• The method provides a framework to allocate learning resources more effectively, such as integrating with prioritized experience replay, by identifying high-value state clusters.