[Paper Review] DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning

Théo Jaunet, Romain Vuillemot+1 more|arXiv (Cornell University)|Sep 6, 2019

Explainable Artificial Intelligence (XAI)68 references36 citations

TL;DR

DRLViz is a visual analytics interface that interprets a DRL agent's internal memory by visualizing a memory timeline, actions, inputs, and derived metrics, enabling memory reduction and expert inquiry in ViZDoom navigation tasks.

ABSTRACT

We present DRLViz, a visual analytics interface to interpret the internal memory of an agent (e.g. a robot) trained using deep reinforcement learning. This memory is composed of large temporal vectors updated when the agent moves in an environment and is not trivial to understand due to the number of dimensions, dependencies to past vectors, spatial/temporal correlations, and co-correlation between dimensions. It is often referred to as a black box as only inputs (images) and outputs (actions) are intelligible for humans. Using DRLViz, experts are assisted to interpret decisions using memory reduction interactions, and to investigate the role of parts of the memory when errors have been made (e.g. wrong direction). We report on DRLViz applied in the context of video games simulators (ViZDoom) for a navigation scenario with item gathering tasks. We also report on experts evaluation using DRLViz, and applicability of DRLViz to other scenarios and navigation problems beyond simulation games, as well as its contribution to black box models interpretability and explainability in the field of visual analytics.

Motivation & Objective

• Explain the internal memory of DRL agents to support interpretability and post-hoc analysis.
• Provide interactive visualizations to browse, filter, and reduce memory for decision analysis.
• Link memory dynamics to agent decisions, inputs, and derived metrics across time and space.
• Validate the approach with domain experts and assess applicability to navigation tasks beyond simulation.

Proposed method

• Use an A2C-based DRL model with RGB inputs and a GRU memory of 128 dimensions.
• Construct a memory matrix by recording hidden states h_t over time (t steps) to form a 2D heat-map memory timeline.
• Expose multiple coordinated visualizations (memory timeline, 2D trajectory map, t-SNE of time-steps, and derived metrics) to relate memory to decisions.
• Provide memory-reduction techniques via re-ordering criteria (activation, change, stable, similar) and by selecting memory intervals.
• Define derived metrics (e.g., health in FoV, item in FoV, orientation variation, decision ambiguity) derived from simulator data for contextual alignment with memory.
• Implement a data pipeline that outputs episode-based JSON data (images, action probabilities, memory vectors, etc.) and a Flask+D3 front-end for visualization.

Experimental results

Research questions

• RQ1 How can experts interpret the connections between a DRL agent's latent memory and its decisions?
• RQ2 Can DRLViz help identify which memory elements influence specific decisions across time and space?
• RQ3 Does memory reduction via interactive filtering and ordering reveal meaningful patterns without destroying interpretability?
• RQ4 What relationships emerge between derived metrics (e.g., items in FoV, orientation variation) and memory activations or decisions?

Key findings

• Experts could identify specific intervals where the agent exhibited repetitive turning and dead-ends using the memory timeline and action views.
• A hidden state dimension was observed to correlate with the sequence of seen items (e.g., red and green armor) and remained active across later steps.
• Users probed memory elements that stayed active when a health pack was first seen and remained active through collection.
• Saliency maps helped experts understand which inputs the agent ignored (e.g., soul-sphere until key items gathered).
• t-SNE clusters in memory revealed objective-aligned groupings (e.g., strategies like gathering green armor), demonstrating meaningful structure in the hidden state space.
• Overall, DRLViz demonstrated potential to interpret DRL memory and suggested applicability to other scenarios beyond ViZDoom.