[Paper Review] Visual Diagnostics for Deep Reinforcement Learning Policy Development

Jieliang Luo, Sam Green+3 more|arXiv (Cornell University)|Sep 14, 2018

Reinforcement Learning in Robotics7 references3 citations

TL;DR

This paper introduces visual diagnostic techniques—class visualization, attribution mapping, and t-SNE—adapted for deep reinforcement learning (DRL) policies in vision-based control. By applying these methods to a simulated drone environment, the authors reveal how policies make decisions, expose biases, and identify flaws such as overfitting to early experiences or poor generalization, significantly improving interpretability and trust in DRL systems.

ABSTRACT

Modern vision-based reinforcement learning techniques often use convolutional neural networks (CNN) as universal function approximators to choose which action to take for a given visual input. Until recently, CNNs have been treated like black-box functions, but this mindset is especially dangerous when used for control in safety-critical settings. In this paper, we present our extensions of CNN visualization algorithms to the domain of vision-based reinforcement learning. We use a simulated drone environment as an example scenario. These visualization algorithms are an important tool for behavior introspection and provide insight into the qualities and flaws of trained policies when interacting with the physical world. A video may be seen at https://sites.google.com/view/drlvisual .

Motivation & Objective

• To address the opacity of convolutional neural networks (CNNs) in vision-based deep reinforcement learning (DRL), especially in safety-critical cyber-physical systems.
• To adapt existing CNN visualization techniques—t-SNE, class visualization, and attribution—for use in DRL policy analysis.
• To enable engineers to introspect and debug DRL policies by visualizing what features the policy attends to and why it selects specific actions.
• To identify deficiencies such as overfitting to early experiences or poor generalization through visual diagnostics.
• To improve trust and interpretability in DRL by making policy decision-making processes transparent and analyzable.

Proposed method

• Extended AirSim, a drone simulator, to support reinforcement learning tasks and collect state-action trajectories for policy analysis.
• Applied t-SNE to cluster state observations based on policy action outputs, revealing decision boundaries and policy behavior clusters.
• Used class visualization to generate synthetic images that maximize the probability of specific actions (left, forward, right), showing what visual patterns the policy associates with each action.
• Employed attribution visualization to highlight image regions most influential in determining the policy’s action choice, using saliency maps to indicate feature importance.
• Trained policies using the REINFORCE algorithm with adjustable learning rates to study the impact of training dynamics on policy behavior.
• Visualized stochastic policies by analyzing action probability distributions and their sensitivity to visual input changes.

Experimental results

Research questions

• RQ1 How can existing CNN visualization techniques be adapted to analyze the decision-making processes of vision-based deep reinforcement learning policies?
• RQ2 What visual patterns do high-performing and poor-performing DRL policies learn, and how do they differ in their perception of the environment?
• RQ3 To what extent do DRL policies exhibit biases toward specific visual features, such as occlusions or background patterns, and how can these be diagnosed?
• RQ4 How does the learning rate during policy training affect the emergence of action biases, and can visualization detect early overfitting?
• RQ5 Can visual diagnostics reveal whether a policy is making decisions based on meaningful environmental cues or spurious correlations?

Key findings

• High-performance policies generate class visualizations that clearly associate actions with spatial positions of cubes—e.g., 'left' action is triggered when a cube is on the left side of the camera view.
• Poor-performance policies produce class visualizations where all actions are triggered by noise, indicating a lack of meaningful feature learning and poor generalization.
• The forward-and-right-only policy shows that high learning rates can cause premature convergence, eliminating 'left' action probabilities despite their relevance.
• Attribution visualizations for high-performing policies show that the closest cube is most salient, confirming that the policy attends to the most relevant environmental features.
• In the poor-performance policy, attribution maps show uniform attention across the image, indicating no reliable feature prioritization and poor decision-making.
• In the forward-and-right-only policy, attribution maps reveal that the policy pays more attention to the horizon than to cubes, suggesting a failure to attend to relevant objects.