[Paper Review] Transparency and Explanation in Deep Reinforcement Learning Neural Networks

Rahul Iyer, Yuezhang Li+4 more|D-Scholarship@Pitt (University of Pittsburgh)|Sep 17, 2018

Explainable Artificial Intelligence (XAI)23 references16 citations

TL;DR

This paper proposes an Object-sensitive Deep Reinforcement Learning (O-DRL) framework that integrates explicit object recognition and valence (positive/negative influence) into DRL models to generate interpretable object saliency maps. These maps visualize which game objects drive policy decisions, enabling human-understandable explanations; human experiments show they are as effective as raw screenshots for predicting agent behavior, especially in complex scenarios with ambiguous rewards.

ABSTRACT

Autonomous AI systems will be entering human society in the near future to provide services and work alongside humans. For those systems to be accepted and trusted, the users should be able to understand the reasoning process of the system, i.e. the system should be transparent. System transparency enables humans to form coherent explanations of the system's decisions and actions. Transparency is important not only for user trust, but also for software debugging and certification. In recent years, Deep Neural Networks have made great advances in multiple application areas. However, deep neural networks are opaque. In this paper, we report on work in transparency in Deep Reinforcement Learning Networks (DRLN). Such networks have been extremely successful in accurately learning action control in image input domains, such as Atari games. In this paper, we propose a novel and general method that (a) incorporates explicit object recognition processing into deep reinforcement learning models, (b) forms the basis for the development of "object saliency maps", to provide visualization of internal states of DRLNs, thus enabling the formation of explanations and (c) can be incorporated in any existing deep reinforcement learning framework. We present computational results and human experiments to evaluate our approach.

Motivation & Objective

• To address the opacity of deep reinforcement learning (DRL) agents, which hinders user trust and system debugging.
• To enable autonomous agents to provide human-understandable explanations of their decisions by incorporating object-level features into the learning process.
• To develop a generalizable method that enhances transparency without requiring architectural overhaul of existing DRL frameworks.
• To evaluate the effectiveness of object saliency maps in improving human understanding of DRL agent behavior through computational and human experiments.

Proposed method

• Integrate explicit object recognition and object valence (positive/negative reward influence) into the DRL network architecture.
• Modify existing DRL frameworks (e.g., DQN, A3C) by adding a branch that processes object features and their saliency in the state representation.
• Train the O-DRL model end-to-end using standard DRL loss functions while preserving the ability to backpropagate saliency scores.
• Generate object saliency maps by computing gradients of the Q-value or policy output with respect to object features, highlighting which objects most influence decisions.
• Use these saliency maps to visualize the relative importance of objects (e.g., dots, ghosts, cherries) in Ms. Pac-Man, showing both presence and valence.
• Apply the method to Atari games like Ms. Pac-Man, where objects have distinct reward implications, to test interpretability and human comprehension.

Experimental results

Research questions

• RQ1 Can object-aware DRL models improve transparency by making internal decision-making processes visible to humans?
• RQ2 How effective are object saliency maps in helping humans predict DRL agent behavior compared to raw game screenshots?
• RQ3 In complex decision scenarios with multiple influencing objects, do saliency maps provide clearer insights than visual frames alone?
• RQ4 Can object saliency maps help identify and explain suboptimal or unexpected behaviors in DRL agents?

Key findings

• Object saliency maps were as effective as raw game screenshots in enabling human participants to predict the behavior of a DRL agent in Ms. Pac-Man.
• In complex scenarios—such as when the agent faced a choice between moving left or down—60% of participants using saliency maps correctly predicted the agent’s action based on object valence (e.g., dots below).
• In contrast, 75% of participants using screenshots predicted the agent would turn left, citing the cherry on the left, indicating that saliency maps revealed decision drivers not obvious from visual frames alone.
• The results suggest that object saliency maps can expose the true decision-making rationale behind DRL policies, especially when reward structures lead to non-intuitive behavior.
• Performance differences across trials indicate that saliency maps are particularly valuable in ambiguous or high-cognitive-load scenarios where visual context is insufficient.
• The method is generalizable and can be integrated into existing DRL frameworks like DQN and A3C without requiring major architectural changes.