[Paper Review] Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Tilman Räuker, Anson Ho+2 more|arXiv (Cornell University)|Jul 27, 2022

Adversarial Robustness in Machine Learning15 citations

TL;DR

This survey provides a comprehensive taxonomy of inner interpretability methods for deep neural networks (DNNs), categorizing techniques by the network component they explain (weights, neurons, subnetworks, latent representations) and whether they are intrinsic or post hoc. It identifies critical connections between interpretability and areas like adversarial robustness, continual learning, and modularity, while advocating for diagnostic tools, benchmarking, and adversarial testing to improve practical utility in engineering contexts.

ABSTRACT

The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to identify problems, fix bugs, and improve basic understanding. In particular, "inner" interpretability techniques, which focus on explaining the internal components of DNNs, are well-suited for developing a mechanistic understanding, guiding manual modifications, and reverse engineering solutions. Much recent work has focused on DNN interpretability, and rapid progress has thus far made a thorough systematization of methods difficult. In this survey, we review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain (weights, neurons, subnetworks, or latent representations) and whether they are implemented during (intrinsic) or after (post hoc) training. To our knowledge, we are also the first to survey a number of connections between interpretability research and work in adversarial robustness, continual learning, modularity, network compression, and studying the human visual system. We discuss key challenges and argue that the status quo in interpretability research is largely unproductive. Finally, we highlight the importance of future work that emphasizes diagnostics, debugging, adversaries, and benchmarking in order to make interpretability tools more useful to engineers in practical applications.

Motivation & Objective

• To systematize and survey over 300 works on inner interpretability in deep neural networks.
• To address the lack of rigorous evaluation and diagnostic tools in current interpretability research.
• To identify and highlight connections between interpretability and key deep learning topics such as adversarial robustness, continual learning, and network compression.
• To argue that current interpretability practices are largely unproductive and call for a shift toward engineering-focused diagnostics and benchmarking.
• To promote the development of tools that support debugging, adversarial testing, and mechanistic understanding in real-world AI deployment.

Proposed method

• Proposes a dual-axis taxonomy: (1) the network component being explained (weights, neurons, subnetworks, latent representations), and (2) the timing of application (intrinsic vs. post hoc).
• Classifies methods based on their target—e.g., neuron activation analysis, weight sparsity, subnetwork interpretation, or representation disentanglement.
• Integrates interpretability with other deep learning paradigms, such as continual learning (via task-specific weight specialization) and adversarial robustness (via interpretability-driven defense design).
• Emphasizes the use of interpretability for diagnostics, including failure analysis, bias detection, and adversarial example probing.
• Proposes a shift from purely explanatory tools to diagnostic tools that support debugging and verification in practice.
• Introduces the concept of 'microscope AI'—using interpretability to reverse-engineer and understand superhuman model behavior.

Figure 1: A taxonomy of inner interpretability methods and hazards associated with them. This mirrors our organization of Sections II - V . We organize methods first by what part of the network’s computational graph they help to explain: weights , neurons , subnetworks , or latent representations .

Experimental results

Research questions

• RQ1 How can interpretability methods be systematically categorized based on the network component they explain and their deployment timing?
• RQ2 What are the key connections between interpretability and other deep learning research areas such as adversarial robustness, continual learning, and modularity?
• RQ3 Why is the current state of interpretability research considered unproductive for real-world AI engineering?
• RQ4 How can interpretability tools be made more useful for debugging, adversarial testing, and model diagnostics in practice?
• RQ5 What role does interpretability play in enabling mechanistic understanding and reverse engineering of DNNs, especially in high-performance models?

Key findings

• The survey identifies over 300 works on inner interpretability, establishing a comprehensive taxonomy that classifies methods by target component and timing (intrinsic/post hoc).
• Interpretability is deeply connected to adversarial robustness, as interpretability tools can reveal vulnerabilities and guide robust model design.
• Continual learning and modular network design benefit from interpretability, particularly through task-specific weight specialization and neuron role identification.
• Network compression and pruning techniques are enhanced when guided by interpretability, as shown by methods like 'pruning by explaining' that use activation-based criteria.
• Post-hoc interpretability methods, such as concept bottleneck models and attention visualization, are widely used but often lack diagnostic power for real-world debugging.
• The paper argues that current interpretability research lacks diagnostic rigor and calls for benchmarking, adversarial testing, and engineering-focused evaluation to improve practical utility.

Figure 2: Inner interpretability methods for weights can focus on (1) continual learning techniques that make weights specialize in particular tasks or (2) weight-masking techniques which learn a mask over weights as a way of discovering which weights are key for a certain task.