[Paper Review] A Detailed Study of Interpretability of Deep Neural Network based Top Taggers

Ayush Khot, M. S. Neubauer+1 more|arXiv (Cornell University)|Oct 9, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper presents a comprehensive analysis of interpretability in deep neural network-based top taggers for high-energy physics, evaluating multiple XAI methods to identify critical features and latent representations in jet classification. It reveals limitations in existing explainability techniques, proposes the Particle Flow Interaction Network (PFIN) model with physics-informed feature engineering, and demonstrates improved performance through interpretability-driven model augmentation, achieving enhanced robustness and clarity in DNN decision-making for top quark jet identification at the LHC.

ABSTRACT

Recent developments in the methods of explainable AI (XAI) allow researchers to explore the inner workings of deep neural networks (DNNs), revealing crucial information about input-output relationships and realizing how data connects with machine learning models. In this paper we explore interpretability of DNN models designed to identify jets coming from top quark decay in high energy proton-proton collisions at the Large Hadron Collider (LHC). We review a subset of existing top tagger models and explore different quantitative methods to identify which features play the most important roles in identifying the top jets. We also investigate how and why feature importance varies across different XAI metrics, how correlations among features impact their explainability, and how latent space representations encode information as well as correlate with physically meaningful quantities. Our studies uncover some major pitfalls of existing XAI methods and illustrate how they can be overcome to obtain consistent and meaningful interpretation of these models. We additionally illustrate the activity of hidden layers as Neural Activation Pattern (NAP) diagrams and demonstrate how they can be used to understand how DNNs relay information across the layers and how this understanding can help to make such models significantly simpler by allowing effective model reoptimization and hyperparameter tuning. These studies not only facilitate a methodological approach to interpreting models but also unveil new insights about what these models learn. Incorporating these observations into augmented model design, we propose the Particle Flow Interaction Network (PFIN) model and demonstrate how interpretability-inspired model augmentation can improve top tagging performance.

Motivation & Objective

• To investigate the interpretability of deep neural network-based top taggers used in high-energy physics for identifying top quark jets at the LHC.
• To evaluate and compare multiple XAI methods in identifying the most influential input features and their roles in model predictions.
• To analyze how feature correlations and latent space representations affect model explainability and performance.
• To identify pitfalls in existing XAI techniques and propose solutions for consistent, meaningful interpretation of DNNs in particle physics.
• To develop and validate a new model architecture, the Particle Flow Interaction Network (PFIN), using interpretability-driven design to improve top tagging accuracy and model simplicity.

Proposed method

• The study evaluates a subset of existing top tagger models, focusing on their feature importance using multiple XAI metrics such as integrated gradients, saliency maps, and activation-based methods.
• It introduces Neural Activation Pattern (NAP) diagrams to visualize information flow across hidden layers, enabling analysis of how features propagate and transform through the network.
• The authors design the Particle Flow Interaction Network (PFIN), which incorporates physics-inspired edge features—ΔR, kT, z, and m²—between jet constituents to enhance interpretability and model performance.
• PFIN uses trainable transformation networks (ΦI,2) to embed particle-particle interactions and concatenate them with original particle features, followed by aggregation into jet-level latent representations.
• The model employs a final classifier network (F) that predicts jet class probabilities based on the aggregated jet-level features, ensuring permutation invariance and physical consistency.
• The study uses both synthetic and real LHC jet datasets to validate XAI methods and assess model performance, comparing PFIN against baseline DNNs.

Experimental results

Research questions

• RQ1 How do different XAI metrics compare in identifying the most important features for top jet classification, and why do they yield inconsistent results?
• RQ2 How do correlations among jet constituents affect the reliability and consistency of feature importance scores in DNNs?
• RQ3 To what extent do latent space representations in DNNs encode physically meaningful quantities such as jet substructure or parton shower history?
• RQ4 Can interpretability-driven model design lead to improved performance and simplified architectures in top tagging?
• RQ5 How can Neural Activation Pattern (NAP) diagrams be used to guide model reoptimization and hyperparameter tuning?

Key findings

• Different XAI methods produce inconsistent feature importance rankings, revealing that model-agnostic methods can be misleading due to feature correlation and gradient saturation.
• Highly correlated features often lead to spurious importance scores, and the study identifies specific failure modes in popular XAI techniques like integrated gradients and saliency maps.
• Latent space representations in DNNs show strong correlations with physically meaningful quantities such as jet mass and substructure, validating their interpretability.
• The PFIN model, incorporating physics-inspired interaction features (ΔR, kT, z, m²), achieves superior performance over baseline DNNs, demonstrating that interpretability-driven design enhances both accuracy and robustness.
• NAP diagrams effectively reveal information flow across layers, enabling targeted reoptimization and hyperparameter tuning that simplify the model while maintaining or improving performance.
• The study demonstrates that augmenting input features with physics-informed interaction embeddings leads to more stable and interpretable DNNs, reducing reliance on black-box behavior.