[Paper Review] Interpretable Diffusion via Information Decomposition

Xianghao Kong, Ollie Liu+3 more|arXiv (Cornell University)|Oct 12, 2023

Model Reduction and Neural NetworksPhysics and Astronomy3 citations

TL;DR

This paper introduces a novel information-theoretic framework that leverages denoising diffusion models to enable fine-grained, interpretable decomposition of information flows between text and images. By deriving exact expressions for mutual information and conditional mutual information (CMI) from the denoising model, the method enables per-pixel and per-word interpretability, outperforming attention mechanisms in detecting the effects of prompt interventions and measuring compositional understanding.

ABSTRACT

Denoising diffusion models enable conditional generation and density modeling of complex relationships like images and text. However, the nature of the learned relationships is opaque making it difficult to understand precisely what relationships between words and parts of an image are captured, or to predict the effect of an intervention. We illuminate the fine-grained relationships learned by diffusion models by noticing a precise relationship between diffusion and information decomposition. Exact expressions for mutual information and conditional mutual information can be written in terms of the denoising model. Furthermore, pointwise estimates can be easily estimated as well, allowing us to ask questions about the relationships between specific images and captions. Decomposing information even further to understand which variables in a high-dimensional space carry information is a long-standing problem. For diffusion models, we show that a natural non-negative decomposition of mutual information emerges, allowing us to quantify informative relationships between words and pixels in an image. We exploit these new relations to measure the compositional understanding of diffusion models, to do unsupervised localization of objects in images, and to measure effects when selectively editing images through prompt interventions.

Motivation & Objective

• To address the lack of interpretability in diffusion models, which obscure precisely which image-text relationships are learned.
• To develop a method that quantifies fine-grained information flow at the pixel and word level without requiring architectural access.
• To provide a black-box compatible alternative to attention-based interpretability that better captures the true effect of prompt interventions.
• To evaluate and improve compositional understanding in diffusion models using information-theoretic metrics.
• To enable unsupervised localization of abstract words (e.g., adjectives, verbs) in images through information decomposition.

Proposed method

• The method derives exact expressions for mutual information (MI) and conditional mutual information (CMI) using the optimal denoising model in diffusion processes.
• It leverages the MMSE denoising property under Gaussian noise to express information quantities directly from the denoiser's output.
• Pointwise estimates of MI and CMI are computed at the pixel and word level, enabling per-sample and per-variable interpretability.
• The framework is applied to analyze prompt interventions, where CMI reflects actual output changes, unlike attention maps.
• The approach is architecture-agnostic, enabling use with black-box APIs, unlike attention-based methods requiring model access.
• Information decomposition is used to quantify compositional understanding and localize semantic concepts in images without supervision.

Experimental results

Research questions

• RQ1 How can we interpret the fine-grained relationships between specific words and image pixels in diffusion models?
• RQ2 To what extent does conditional mutual information (CMI) better reflect the true effect of prompt interventions than attention mechanisms?
• RQ3 Can information-theoretic measures be used to evaluate compositional generalization in text-to-image diffusion models?
• RQ4 How do information decomposition and attention compare in localizing abstract linguistic terms like adverbs and adjectives in images?
• RQ5 Can CMI estimates serve as a reliable metric for identifying model circuits or functional components in diffusion models?

Key findings

• The proposed method enables exact, tractable computation of mutual information and conditional mutual information directly from the denoising model, providing a principled information-theoretic interpretation of diffusion models.
• CMI estimates correctly identify when a prompt intervention has no effect on the generated image, whereas attention maps fail to reflect this, demonstrating CMI's superiority as a dependence measure.
• On the ARO benchmark, diffusion models show improved compositional understanding over their contrastive VLM backbones, but still underperform compared to models pre-trained with composition-aware negatives.
• Information decomposition localizes abstract words (e.g., adjectives, verbs) more effectively than attention, even without supervision, due to its sensitivity to contextual dependencies.
• The method reveals inconsistent changes in mutual information during prompt interventions, suggesting potential dataset idiosyncrasies or model sensitivity to input context.
• The framework is adaptable to high-stakes domains such as biomedical modeling and protein design, where identifying informative variables is critical.