[Paper Review] Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?

Matthias Rosynski, Frank Kirchner+1 more|arXiv (Cornell University)|Dec 2, 2020

Reinforcement Learning in Robotics14 references4 citations

TL;DR

This paper investigates the utility of gradient-based saliency maps in Deep Reinforcement Learning (DRL), demonstrating that techniques like Grad-CAM and Guided Backpropagation provide early, interpretable insights into agent decision-making—outperforming rewards alone for debugging. It introduces G1Grad-Cam, a novel method that detects training errors early and reveals feature importance more effectively than standard Grad-CAM, especially in off-policy algorithms where rewards are initially unreliable.

ABSTRACT

Deep Reinforcement Learning (DRL) connects the classic Reinforcement Learning algorithms with Deep Neural Networks. A problem in DRL is that CNNs are black-boxes and it is hard to understand the decision-making process of agents. In order to be able to use RL agents in highly dangerous environments for humans and machines, the developer needs a debugging tool to assure that the agent does what is expected. Currently, rewards are primarily used to interpret how well an agent is learning. However, this can lead to deceptive conclusions if the agent receives more rewards by memorizing a policy and not learning to respond to the environment. In this work, it is shown that this problem can be recognized with the help of gradient visualization techniques. This work brings some of the best-known visualization methods from the field of image classification to the area of Deep Reinforcement Learning. Furthermore, two new visualization techniques have been developed, one of which provides particularly good results. It is being proven to what extent the algorithms can be used in the area of Reinforcement learning. Also, the question arises on how well the DRL algorithms can be visualized across different environments with varying visualization techniques.

Motivation & Objective

• To assess whether gradient-based saliency maps can serve as effective debugging tools in Deep Reinforcement Learning.
• To evaluate the interpretability and reliability of existing visualization techniques—such as Grad-CAM and Guided Backpropagation—within DRL environments.
• To develop and validate new visualization methods, particularly G1Grad-Cam, that enhance feature detection and error localization during training.
• To determine whether saliency methods can detect architectural or implementation errors earlier than reward-based analysis, especially in off-policy algorithms.
• To investigate whether action-discriminative saliency methods can highlight features most influential in the agent’s decision-making process.

Proposed method

• Adapted and applied four established gradient-based visualization techniques—Grad-CAM, Guided Backpropagation, Grad-CAM++, and a novel method G1Grad-Cam—to DRL agents in Atari environments.
• Proposed G1Grad-Cam as a modified Grad-CAM variant that improves feature localization by emphasizing high-gradient regions in the final convolutional layer.
• Used perturbation-based saliency metrics (e.g., finite difference approximation of gradients) to compare the impact of input region removal on policy output.
• Applied visualization techniques across multiple DRL agents (e.g., DDDQN, Split Attention Agent) in environments like Breakout and Pong to assess consistency and interpretability.
• Evaluated saliency maps at early training stages, comparing them with sparse and noisy reward signals to assess early diagnostic potential.
• Performed ablation studies to analyze the influence of negative gradients and their role in highlighting feature boundaries, suggesting a Laplacian-like edge detection effect.

Experimental results

Research questions

• RQ1 Can gradient-based saliency maps detect training errors earlier than reward-based monitoring in off-policy DRL algorithms?
• RQ2 Do Guided Backpropagation and Grad-CAM methods reveal meaningful, physically interpretable features in DRL agents, or do they produce misleading artifacts?
• RQ3 Can the proposed G1Grad-Cam method better identify features that influence the agent’s action compared to standard Grad-CAM?
• RQ4 To what extent do saliency maps reveal whether an agent is learning the correct features or memorizing patterns due to reward shaping?
• RQ5 Is there a way to make saliency methods action-discriminative, so they highlight features most responsible for the selected action?

Key findings

• G1Grad-Cam and Grad-CAM produced interpretable visualizations very early in training, even before rewards began to increase, enabling rapid detection of network issues.
• Guided Backpropagation outperformed Grad-CAM in detecting feature importance and identifying errors in specific network layers or streams, particularly in the Split Attention Agent.
• The method successfully visualized a frame-stacking error in the DDDQN agent within minutes, allowing immediate correction before extensive training.
• Negative gradients in Guided Backpropagation formed patterns resembling a Laplacian kernel, suggesting they detect feature boundaries through second-derivative effects.
• Despite claims in prior work, Guided Backpropagation was found to visualize relevant features rather than acting as a mere edge detector, challenging earlier skepticism.
• No action-discriminative saliency method in this study clearly highlighted features with the greatest influence on the selected action, indicating a need for future improvements in action-specific gradient routing.