[论文解读] Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Tilman Räuker, Anson Ho+2 more|arXiv (Cornell University)|Jul 27, 2022

Adversarial Robustness in Machine Learning被引用 15

一句话总结

本综述系统地梳理了深度神经网络（DNNs）内部可解释性方法的全面分类体系，按其解释的网络组件（权重、神经元、子网络、潜在表征）以及方法是内在（intrinsic）还是事后（post hoc）进行分类。该综述揭示了可解释性与对抗鲁棒性、持续学习、模块化设计等关键领域的深层联系，同时倡导开发诊断工具、基准测试以及对抗性测试，以提升在工程实践中的实用性。

ABSTRACT

The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to identify problems, fix bugs, and improve basic understanding. In particular, "inner" interpretability techniques, which focus on explaining the internal components of DNNs, are well-suited for developing a mechanistic understanding, guiding manual modifications, and reverse engineering solutions. Much recent work has focused on DNN interpretability, and rapid progress has thus far made a thorough systematization of methods difficult. In this survey, we review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain (weights, neurons, subnetworks, or latent representations) and whether they are implemented during (intrinsic) or after (post hoc) training. To our knowledge, we are also the first to survey a number of connections between interpretability research and work in adversarial robustness, continual learning, modularity, network compression, and studying the human visual system. We discuss key challenges and argue that the status quo in interpretability research is largely unproductive. Finally, we highlight the importance of future work that emphasizes diagnostics, debugging, adversaries, and benchmarking in order to make interpretability tools more useful to engineers in practical applications.

研究动机与目标

• 系统化并综述超过300篇关于深度神经网络内部可解释性的研究工作。
• 解决当前可解释性研究中缺乏严格评估与诊断工具的问题。
• 识别并强调可解释性与对抗鲁棒性、持续学习、网络压缩等关键深度学习主题之间的关联。
• 指出当前可解释性实践在很大程度上缺乏成效，并呼吁转向以工程为导向的诊断工具与基准测试。
• 推动开发支持调试、对抗性测试与机制理解的工具，以促进真实世界AI部署中的可解释性。

提出的方法

• 提出双轴分类体系：(1) 所解释的网络组件（权重、神经元、子网络、潜在表征），(2) 应用时间（内在 vs. 事后）。
• 根据目标对方法进行分类，例如神经元激活分析、权重稀疏性分析、子网络解释或表征解耦分析。
• 将可解释性与其它深度学习范式整合，例如通过任务特定权重专精实现持续学习，或通过可解释性驱动的防御设计提升对抗鲁棒性。
• 强调可解释性在诊断中的应用，包括故障分析、偏差检测以及对抗性样本探测。
• 主张从纯解释性工具转向支持实际调试与验证的诊断工具。
• 引入“显微镜AI”概念——利用可解释性反向工程并理解超人类性能模型的行为。

Figure 1: A taxonomy of inner interpretability methods and hazards associated with them. This mirrors our organization of Sections II - V . We organize methods first by what part of the network’s computational graph they help to explain: weights , neurons , subnetworks , or latent representations .

实验结果

研究问题

• RQ1 可解释性方法能否基于其解释的网络组件及其部署时机（内在或事后）实现系统化分类？
• RQ2 可解释性与对抗鲁棒性、持续学习、模块化等其他深度学习研究领域之间存在哪些关键联系？
• RQ3 为何当前可解释性研究状态被认为对真实世界AI工程无实质帮助？
• RQ4 如何使可解释性工具在实际调试、对抗性测试与模型诊断中更具实用性？
• RQ5 可解释性在实现对DNN的机制理解与反向工程中扮演何种角色，特别是在高性能模型中？

主要发现

• 该综述识别出超过300篇关于内部可解释性的研究，建立了全面的分类体系，按目标组件与应用时机（内在/事后）对方法进行分类。
• 可解释性与对抗鲁棒性密切相关，因为可解释性工具可揭示模型漏洞，并指导更具鲁棒性的模型设计。
• 持续学习与模块化网络设计得益于可解释性，特别是通过任务特定权重专精与神经元角色识别。
• 当由可解释性引导时，网络压缩与剪枝技术得到增强，例如‘通过解释进行剪枝’方法利用基于激活的准则实现更优效果。
• 事后可解释性方法（如概念瓶颈模型与注意力可视化）虽广泛应用，但常缺乏对真实世界调试的诊断能力。
• 本文认为当前可解释性研究缺乏诊断严谨性，呼吁引入基准测试、对抗性测试与面向工程的评估，以提升实际应用价值。

Figure 2: Inner interpretability methods for weights can focus on (1) continual learning techniques that make weights specialize in particular tasks or (2) weight-masking techniques which learn a mask over weights as a way of discovering which weights are key for a certain task.