[Paper Review] Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

Nikaash Puri, Sukriti Verma+5 more|arXiv (Cornell University)|Dec 23, 2019

Explainable Artificial Intelligence (XAI)27 references32 citations

TL;DR

SARFA introduces a perturbation-based saliency method that emphasizes action-specific and relevant features, producing more interpretable explanations of RL agent actions than prior approaches across chess, Go, and Atari. It combines specificity (impact on the chosen action) and relevance (minimizing impact on other actions) using a harmonic mean.

ABSTRACT

As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents. Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action. Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent. Our proposed approach, SARFA (Specific and Relevant Feature Attribution), generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency. The first captures the impact of perturbation on the relative expected reward of the action to be explained. The second downweighs irrelevant features that alter the relative expected rewards of actions other than the action to be explained. We compare SARFA with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders). We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches. For the code release and demo videos, see https://nikaashpuri.github.io/sarfa-saliency/.

Motivation & Objective

• Motivate interpretable explanations for deep reinforcement learning agents as they are applied to board and arcade games.
• Develop a saliency method that focuses on features relevant to the specific action chosen by the agent.
• Address limitations of prior perturbation-based saliency methods that highlight non-relevant features or uniform action effects.

Proposed method

• Define saliency S[f] for each state feature f based on perturbations and Q-values.
• Use softmax over Q-values to compute relative returns P(s, ‘‘a’’) and the perturbation impact Δp on the selected action.
• Compute a relevance term via KL-divergence between normalized relative returns excluding the chosen action.
• Combine Δp and the similarity K = 1/(1 + D_KL) with a harmonic mean S[f] = 2KΔp/(K+Δp).
• Ensure saliency highlights features that specifically affect the chosen action while downweighting features that affect other actions.
• Evaluate SARFA on Chess (Stockfish), Go (MiniGo), and Atari (Breakout, Pong, Space Invaders) under black-box Q(s, a) access.

Experimental results

Research questions

• RQ1 Does SARFA produce more action-focused and human-interpretable saliency maps than prior perturbation-based methods?
• RQ2 How do specificity and relevance contribute to improved human understanding in Chess, Go, and Atari domains?
• RQ3 Can SARFA reveal meaningful tactical motifs in chess and improve puzzle-solving performance for humans?
• RQ4 Is SARFA robust to perturbations and applicable to black-box RL agents?

Key findings

• SARFA yields more focused saliency maps that highlight pieces or regions actually relevant to the chosen move or action.
• In chess puzzles with human participants, SARFA saliency led to higher accuracy (72.41%) and faster solving times (67.02 s) compared with baselines.
• SARFA’s chess dataset ROC analysis shows it better identifies human-relevant pieces than Greydanus et al. and Iyer et al. approaches.
• Humans solving chess puzzles with SARFA guidance performed roughly 25% more accurately and 31% faster than baselines in some setups.
• SARFA demonstrates intuitive explanations of tactical motifs (e.g., pin, mate in 2, overloading) through saliency maps.
• Robustness tests indicate SARFA saliency remains stable under perturbations that do not alter the target action (AUC ~0.92 on perturbed datasets).