[Paper Review] SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning

Ahmed Salem, Giovanni Cherubin+6 more|arXiv (Cornell University)|Dec 21, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces a unified game-based framework to systematize data inference privacy risks in machine learning, formalizing five core threats—membership, attribute, property, differential privacy distinguishability, and data reconstruction inference—as probabilistic games. It establishes rigorous reductions and separations between these risks, revealing previously unknown relationships and enabling clearer reasoning, composition, and mechanization of privacy proofs in ML systems.

ABSTRACT

Deploying machine learning models in production may allow adversaries to infer sensitive information about training data. There is a vast literature analyzing different types of inference risks, ranging from membership inference to reconstruction attacks. Inspired by the success of games (i.e., probabilistic experiments) to study security properties in cryptography, some authors describe privacy inference risks in machine learning using a similar game-based style. However, adversary capabilities and goals are often stated in subtly different ways from one presentation to the other, which makes it hard to relate and compose results. In this paper, we present a game-based framework to systematize the body of knowledge on privacy inference risks in machine learning. We use this framework to (1) provide a unifying structure for definitions of inference risks, (2) formally establish known relations among definitions, and (3) to uncover hitherto unknown relations that would have been difficult to spot otherwise.

Motivation & Objective

• To address the fragmentation in the literature on machine learning privacy attacks, where adversary capabilities and goals are inconsistently defined.
• To formalize five fundamental privacy inference risks—membership, attribute, property, differential privacy distinguishability, and data reconstruction—as game-based definitions with clear semantics.
• To establish formal relationships (reductions and separations) between these privacy risks using a quantitative notion of reduction.
• To enable clearer reasoning, composition, and mechanization of privacy proofs through a structured, code-based game framework.
• To demonstrate the framework’s utility through a case study showing that a complex attack scenario can be decomposed into combinations of simpler inference games.

Proposed method

• Decompose game-based privacy definitions into core components: adversary capabilities, dataset and challenge selection, and success measures.
• Represent each of the five privacy risks as a probabilistic game involving a challenger and an adversary interacting through oracles.
• Define a quantitative notion of reduction between privacy properties, allowing formal proof of implications or impossibility.
• Use sequence-of-games transformations with formal semantics to prove reductions and separations, enabling mechanizable reasoning.
• Apply the framework to analyze and decompose a literature-claimed membership inference variant into a combination of membership and property inference games.
• Leverage existing program logic and verification tools (e.g., EasyCrypt) by modeling games as probabilistic programs with unambiguous semantics.

Experimental results

Research questions

• RQ1 How can we unify the diverse and inconsistent formulations of privacy inference attacks in machine learning?
• RQ2 What formal relationships exist between different inference risks—do security guarantees for one imply security for another?
• RQ3 Can we formally establish when one privacy risk is reducible to another, or when they are fundamentally independent?
• RQ4 To what extent can game-based reasoning be used to decompose and analyze complex attack scenarios?
• RQ5 Can the game-based framework support mechanized proofs and formal verification of privacy properties in ML systems?

Key findings

• The paper establishes a complete set of formal reductions and separations between five core privacy inference risks, revealing that membership inference security implies security against attribute and property inference, but not against data reconstruction.
• It proves that differential privacy distinguishability is strictly weaker than membership inference, meaning DP does not imply membership inference resistance.
• The framework reveals that data reconstruction is strictly stronger than membership inference, as there exist systems secure against membership inference but completely insecure against data reconstruction.
• Through a case study, the paper shows that a complex attack scenario previously described as a variant of membership inference can be formally decomposed into a combination of membership and property inference games.
• The authors demonstrate that game-based reasoning enables structured, mechanizable proofs using formal program transformations, suggesting compatibility with tools like EasyCrypt for formal verification.
• The study identifies that continuous probability distributions (e.g., in DP-SGD) pose a challenge for existing mechanization frameworks, which often assume discrete probability spaces.