[Paper Review] RISE: Randomized Input Sampling for Explanation of Black-box Models

Vitali Petsiuk, Abir Das+1 more|arXiv (Cornell University)|Jun 19, 2018

Explainable Artificial Intelligence (XAI)Computer Science207 citations

TL;DR

RISE explains black-box image classifiers by probing with randomly masked inputs to build pixel-wise saliency maps, evaluated with automatic causal metrics (deletion/insertion) and a human-centric pointing game.

ABSTRACT

Deep neural networks are being used increasingly to automate data analysis and decision making, yet their decision-making process is largely unclear and is difficult to explain to the end users. In this paper, we address the problem of Explainable AI for deep neural networks that take images as input and output a class probability. We propose an approach called RISE that generates an importance map indicating how salient each pixel is for the model's prediction. In contrast to white-box approaches that estimate pixel importance using gradients or other internal network state, RISE works on black-box models. It estimates importance empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. We compare our approach to state-of-the-art importance extraction methods using both an automatic deletion/insertion metric and a pointing metric based on human-annotated object segments. Extensive experiments on several benchmark datasets show that our approach matches or exceeds the performance of other methods, including white-box approaches. Project page: http://cs-people.bu.edu/vpetsiuk/rise/

Motivation & Objective

• Motivate the need for explanations of deep neural networks as black-box decision-makers in vision tasks.
• Propose a general, black-box saliency method that does not require access to model internals.
• Quantitatively evaluate saliency quality using automatic causal metrics and human-centered metrics.
• Demonstrate applicability to multiple datasets and base architectures.
• Show that RISE can extend to explanation of image captioning models.

Proposed method

• Generate random binary masks and upsample them to perturb input images via element-wise masking.
• Compute the saliency map as a weighted sum of masks, where weights are the base model’s output scores on masked inputs.
• Estimate the saliency map with Monte Carlo sampling of N random masks as the average of f(I ⊙ M_i) · M_i, scaled by E[M].
• Use bilinear upsampling and random spatial shifts to create smooth, robust masks.
• Evaluate explanations with deletion and insertion metrics that measure how class probability drops or rises as pixels are removed or added according to the saliency map.
• Compare to white-box and other black-box methods across datasets (PASCAL VOC07, MSCOCO2014, ImageNet).

Experimental results

Research questions

• RQ1 Can a fully black-box approach produce reliable pixel-level explanations for image classifiers without access to gradients or internals?
• RQ2 Do randomized input masks combined with learned mask weights provide competitive saliency maps compared to white-box methods?
• RQ3 How do deletion/insertion metrics reflect the causal quality of explanations across architectures and datasets?
• RQ4 Is RISE applicable to captioning models beyond classification tasks?

Key findings

• RISE achieves better deletion and insertion scores than several baselines on ImageNet with ResNet50 and VGG16 (lower deletion and higher insertion are better).
• RISE's deletion score for ResNet50 is 0.1076±0.0005 and insertion 0.7267±0.0006; for VGG16, deletion 0.0980±0.0025 and insertion 0.6663±0.0014.
• RISE outperforms Grad-CAM on ImageNet in the reported metrics, while remaining a black-box method that does not require internals of the base model.
• On VOC MSCOCO, RISE achieves superior pointing game accuracy compared to other black-box methods and is competitive with white-box approaches.
• RISE is extendable to explain image captioning models by evaluating word-level saliency via masked inputs.
• The method is robust across multiple runs (reported as mean±std. over 3 runs).