[Paper Review] On the Privacy Risks of Algorithmic Recourse

Martin Pawelczyk, Himabindu Lakkaraju+1 more|arXiv (Cornell University)|Nov 10, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper introduces counterfactual distance-based membership inference attacks that exploit algorithmic recourse to infer whether a data instance was part of a model's training data. By analyzing the distance between instances and their generated counterfactuals, the authors demonstrate significant privacy leakage—especially in high-dimensional data—outperforming traditional loss-based attacks and revealing a critical, previously unexplored risk in recourse deployment.

ABSTRACT

As predictive models are increasingly being employed to make consequential decisions, there is a growing emphasis on developing techniques that can provide algorithmic recourse to affected individuals. While such recourses can be immensely beneficial to affected individuals, potential adversaries could also exploit these recourses to compromise privacy. In this work, we make the first attempt at investigating if and how an adversary can leverage recourses to infer private information about the underlying model's training data. To this end, we propose a series of novel membership inference attacks which leverage algorithmic recourse. More specifically, we extend the prior literature on membership inference attacks to the recourse setting by leveraging the distances between data instances and their corresponding counterfactuals output by state-of-the-art recourse methods. Extensive experimentation with real world and synthetic datasets demonstrates significant privacy leakage through recourses. Our work establishes unintended privacy leakage as an important risk in the widespread adoption of recourse methods.

Motivation & Objective

• To investigate whether algorithmic recourse methods inadvertently leak private information about training data.
• To identify and formalize a new class of membership inference attacks that exploit counterfactual distances.
• To evaluate the effectiveness of these attacks across diverse real-world and synthetic datasets.
• To understand the conditions under which such privacy leaks are most severe.
• To highlight the urgent need for privacy-preserving recourse generation in real-world ML deployments.

Proposed method

• Proposes a general class of membership inference attacks called counterfactual distance-based attacks, leveraging distances between input instances and their algorithmic counterfactuals.
• Introduces two specific attacks: a threshold-based method and a likelihood ratio test (LRT) that uses counterfactual distances as input.
• Operates under a realistic assumption: the adversary can query the recourse algorithm only once per instance, mimicking real-world user access.
• Employs log-scaled ROC curves and distance distribution comparisons to evaluate attack performance across datasets and model types.
• Uses both synthetic data (with controlled dimensionality and model capacity) and real-world datasets from finance, healthcare, and law to validate the attacks.
• Compares attack performance against baseline methods, including loss-based membership inference attacks that have access to full model gradients and labels.

Experimental results

Research questions

• RQ1 Can an adversary use algorithmic recourse to determine whether a given data instance was part of the model’s training data?
• RQ2 How does the distance between an instance and its counterfactual reveal information about its membership in the training set?
• RQ3 Under what data and model conditions (e.g., high dimensionality, model overfitting) is this privacy leakage most pronounced?
• RQ4 How does the performance of recourse-based attacks compare to state-of-the-art loss-based membership inference attacks?
• RQ5 To what extent does model capacity or feature dimensionality amplify the risk of privacy leakage through recourses?

Key findings

• The proposed counterfactual distance-based attacks significantly outperform standard loss-based membership inference attacks on high-dimensional data, even when the latter have access to full model gradients and labels.
• Attack performance increases with feature dimensionality, as evidenced by the divergence of training and test instance distance distributions in high-dimensional synthetic data.
• On real-world tabular datasets with low to moderate dimensionality (d < 50), attack performance remains moderate, suggesting that dimensionality is a key factor in leakage severity.
• For nonlinear models such as deep neural networks, increasing model capacity amplifies attack success, especially when combined with high feature dimensionality, with the CFD LRT outperforming even loss-based LRTs in high-capacity settings.
• The attacks are most effective when models overfit to training data, confirming that overfitting exacerbates privacy leakage via recourses.
• Empirical results show that the CFD LRT achieves high AUC on high-dimensional synthetic data, demonstrating that counterfactual distances alone can serve as a powerful signal for membership inference.