[Paper Review] Investigating Saturation Effects in Integrated Gradients

Vivek Miglani, Narine Kokhlikyan+3 more|arXiv (Cornell University)|Oct 23, 2020

Explainable Artificial Intelligence (XAI)18 references4 citations

TL;DR

This paper identifies that gradients in saturated regions of the Integrated Gradients (IG) path contribute disproportionately to attributions despite minimal model output change. It proposes Split Integrated Gradients, which restricts integration to unsaturated regions where output changes substantially, resulting in higher model faithfulness and lower noise sensitivity on ImageNet models compared to standard IG.

ABSTRACT

Integrated Gradients has become a popular method for post-hoc model interpretability. De-spite its popularity, the composition and relative impact of different regions of the integral path are not well understood. We explore these effects and find that gradients in saturated regions of this path, where model output changes minimally, contribute disproportionately to the computed attribution. We propose a variant of IntegratedGradients which primarily captures gradients in unsaturated regions and evaluate this method on ImageNet classification networks. We find that this attribution technique shows higher model faithfulness and lower sensitivity to noise com-pared with standard Integrated Gradients. A note-book illustrating our computations and results is available at https://github.com/vivekmig/captum-1/tree/ExpandedIG.

Motivation & Objective

• To investigate the contribution of gradients in saturated versus unsaturated regions along the IG integral path.
• To evaluate whether saturated region gradients distort attribution quality in standard Integrated Gradients.
• To develop a variant of IG that focuses on unsaturated regions to improve faithfulness and robustness.
• To provide practical guidance for users on assessing saturation and interpreting IG results.

Proposed method

• The method introduces Split Integrated Gradients, which restricts the integral path to regions before model output saturation.
• It defines a threshold α* using a percentage ψ (e.g., 90%) of the output range change from baseline to input.
• The integral is computed only over α ∈ [0, α*], excluding saturated regions where F changes minimally.
• The completeness property is approximately preserved since the majority of output change occurs before α*.
• The approach visualizes the model output and gradient magnitude along α to detect saturation and guide path selection.
• It compares attributions from standard IG and Split IG qualitatively and quantitatively on ImageNet models.

Experimental results

Research questions

• RQ1 To what extent do gradients in saturated regions of the IG path contribute to the final attribution?
• RQ2 How does excluding saturated gradients affect the faithfulness and robustness of attribution results?
• RQ3 Can a modified IG path that excludes saturated regions yield more reliable and interpretable attributions?
• RQ4 How does the choice of baseline and model architecture influence saturation effects in IG?

Key findings

• Gradients in saturated regions of the IG path contribute disproportionately to the final attribution, despite minimal model output change.
• Split Integrated Gradients, which restricts integration to unsaturated regions, produces attributions with higher model faithfulness than standard IG.
• Attributions from Split IG show significantly lower sensitivity to small input perturbations compared to standard IG.
• The method maintains approximate completeness by capturing the majority of output change before saturation occurs.
• Visualization of the output and gradient path along α helps identify saturation and guides better attribution interpretation.
• The study recommends users examine the output path and compare attributions from saturated and unsaturated regions to assess attribution quality.