[Paper Review] A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures

Thành Tâm Nguyên, Thanh Trung Huynh+5 more|arXiv (Cornell University)|Mar 31, 2024

Privacy-Preserving Technologies in Data4 citations

TL;DR

This survey provides the first comprehensive analysis of privacy risks in model explanations, categorizing attacks on explanations (e.g., membership inference, de-anonymization) and proposing countermeasures like differential privacy and adversarial training. It establishes a taxonomy of privacy-preserving explanation techniques and identifies critical research gaps in data modalities, theoretical foundations, and real-world evaluation.

ABSTRACT

As the adoption of explainable AI (XAI) continues to expand, the urgency to address its privacy implications intensifies. Despite a growing corpus of research in AI privacy and explainability, there is little attention on privacy-preserving model explanations. This article presents the first thorough survey about privacy attacks on model explanations and their countermeasures. Our contribution to this field comprises a thorough analysis of research papers with a connected taxonomy that facilitates the categorisation of privacy attacks and countermeasures based on the targeted explanations. This work also includes an initial investigation into the causes of privacy leaks. Finally, we discuss unresolved issues and prospective research directions uncovered in our analysis. This survey aims to be a valuable resource for the research community and offers clear insights for those new to this domain. To support ongoing research, we have established an online resource repository, which will be continuously updated with new and relevant findings. Interested readers are encouraged to access our repository at https://github.com/tamlhp/awesome-privex.

Motivation & Objective

• To address the growing gap in research on privacy risks associated with model explanations in explainable AI (XAI).
• To systematically categorize privacy attacks targeting model explanations based on explanation type and attack objective.
• To analyze existing countermeasures, including differential privacy and adversarial training, for mitigating privacy leaks in explanations.
• To identify unresolved challenges such as limited scope in attack types, lack of theoretical understanding of privacy leakage causes, and insufficient evaluation under realistic conditions.
• To provide a foundational taxonomy and research roadmap for future work in privacy-preserving XAI, supported by an open-source repository.

Proposed method

• Conducting a systematic literature review of 240+ papers on privacy in machine learning, XAI, and adversarial attacks to identify privacy risks in model explanations.
• Developing a multi-dimensional taxonomy to classify privacy attacks based on target explanation type (e.g., local, counterfactual, feature importance), attack goal (e.g., membership inference, data reconstruction), and underlying mechanism.
• Analyzing the root causes of privacy leaks in explanations, including model architecture, explanation method design, and data sensitivity.
• Surveying and categorizing defense mechanisms such as differential privacy, adversarial training, and input perturbation tailored to explanation systems.
• Evaluating the effectiveness of privacy-preserving techniques across different explanation types and data modalities (e.g., tabular, image, graph, audio).
• Establishing an open-source, continuously updated repository (https://github.com/tamlhp/awesome-privex) to support ongoing research and community contributions.

Experimental results

Research questions

• RQ1 What are the primary types of privacy attacks targeting model explanations, and how do they differ across explanation types (e.g., LIME, SHAP, counterfactuals)?
• RQ2 What are the underlying causes of privacy leakage in model explanations, and how do model architecture and explanation methodology contribute to these risks?
• RQ3 Which countermeasures—such as differential privacy, adversarial training, or input perturbation—are most effective in preserving privacy while maintaining explanation fidelity?
• RQ4 How do privacy risks vary across different data modalities, including tabular, image, graph, and audio data, and what unique challenges do they present?
• RQ5 What are the key unresolved research challenges in privacy-preserving model explanations, particularly regarding theoretical foundations, real-world evaluation, and broader model coverage?

Key findings

• Membership inference attacks are the most studied privacy threat in model explanations, particularly targeting local and counterfactual explanations.
• Explanation methods like LIME and SHAP can inadvertently leak membership information due to their reliance on local surrogate models and feature perturbations.
• Differential privacy mechanisms, especially adaptive noise injection, can preserve model interpretability while reducing privacy risks, though they may degrade explanation fidelity.
• Graph neural networks (GNNs) are emerging as a high-risk area for privacy leaks, with attacks like membership inference and de-anonymization being increasingly feasible on GNN explanations.
• Audio-based model explanations remain under-researched in terms of privacy, despite high sensitivity of audio data (e.g., voice, health signals) and growing use in healthcare applications.
• There is a significant lack of theoretical understanding of why certain explanations are more prone to privacy leakage, and few studies evaluate attacks under realistic deployment conditions (e.g., large datasets, real-world models).