[Paper Review] Learning to Succeed while Teaching to Fail: Privacy in Closed Machine Learning Systems

Jure Sokolić, Qiang Qiu+2 more|arXiv (Cornell University)|May 23, 2017

Privacy-Preserving Technologies in Data20 references3 citations

TL;DR

This paper proposes a closed-loop machine learning framework that trains models to succeed on a primary task (e.g., medical diagnosis) while actively failing at inferring sensitive attributes (e.g., gender or race), ensuring privacy without relying on universal adversarial assumptions. The method uses constrained linear projection and Maximum Mean Discrepancy-based noise injection to degrade sensitive attribute inference while preserving utility on the main task, achieving 55% accuracy on protected tasks in experiments.

ABSTRACT

Security, privacy, and fairness have become critical in the era of data science and machine learning. More and more we see that achieving universally secure, private, and fair systems is practically impossible. We have seen for example how generative adversarial networks can be used to learn about the expected private training data; how the exploitation of additional data can reveal private information in the original one; and how what looks like unrelated features can teach us about each other. Confronted with this challenge, in this paper we open a new line of research, where the security, privacy, and fairness is learned and used in a closed environment. The goal is to ensure that a given entity (e.g., the company or the government), trusted to infer certain information with our data, is blocked from inferring protected information from it. For example, a hospital might be allowed to produce diagnosis on the patient (the positive task), without being able to infer the gender of the subject (negative task). Similarly, a company can guarantee that internally it is not using the provided data for any undesired task, an important goal that is not contradicting the virtually impossible challenge of blocking everybody from the undesired task. We design a system that learns to succeed on the positive task while simultaneously fail at the negative one, and illustrate this with challenging cases where the positive task is actually harder than the negative one being blocked. Fairness, to the information in the negative task, is often automatically obtained as a result of this proposed approach. The particular framework and examples open the door to security, privacy, and fairness in very important closed scenarios, ranging from private data accumulation companies like social networks to law-enforcement and hospitals.

Motivation & Objective

• To address the growing challenge of privacy leakage in machine learning systems where sensitive attributes (e.g., gender, race) can be inferred from data.
• To design a system that ensures a trusted entity (e.g., hospital or company) can perform a desired task (e.g., diagnosis) without being able to infer protected information.
• To overcome the limitations of universal privacy guarantees by operating in a closed, controlled environment where privacy is enforced through model design rather than external constraints.
• To demonstrate that privacy can be achieved even when the primary task is more complex than the protected task, challenging the assumption that privacy is only feasible when the protected task is easier.
• To provide a practical, empirically validated framework for privacy enforcement that can be applied in real-world systems like healthcare, law enforcement, and social networks.

Proposed method

• The framework uses a linear projection matrix $ P $ that is iteratively updated to collapse the data space along directions that separate sensitive attributes, using the weight vector of a trained SVM on the sensitive label.
• The projection is updated via $ P = P'P $, where $ P' $'s kernel aligns with the SVM’s weight vector, progressively reducing the model’s ability to classify sensitive attributes.
• A Maximum Mean Discrepancy (MMD)-based transformation adds non-linear noise to input features by learning a coefficient vector $ \theta(x) $ that shifts the input toward the distribution of a randomly chosen class (positive or negative) of the sensitive attribute.
• The noise vector $ n(x) = D\theta(x) $ is computed using a Gaussian kernel $ k(x,x') = \exp(-\|x-x'\|_2^2 / 2\sigma^2) $ with $ \sigma = 0.001 $, and optimized via gradient ascent to maximize class similarity.
• The optimization is performed over 10,000 iterations with a learning rate of 0.05, ensuring the transformed data becomes ambiguous with respect to the sensitive attribute.
• The trade-off between utility and privacy is controlled by adjusting the dimensionality of the projection kernel or the magnitude of the MMD-based perturbation.

Experimental results

Research questions

• RQ1 Can a machine learning system be designed to succeed on a primary task while being explicitly prevented from inferring sensitive attributes, even when the primary task is more complex?
• RQ2 How can privacy be enforced in a closed system where the data processor is trusted but must not access or infer protected information?
• RQ3 To what extent can linear projection and MMD-based transformations degrade sensitive attribute inference without degrading performance on the primary task?
• RQ4 Is fairness with respect to sensitive attributes automatically achieved when the system is trained to fail on those attributes?
• RQ5 Can this framework provide practical, empirically verifiable privacy guarantees in real-world applications such as healthcare and social media?

Key findings

• The proposed linear projection method successfully reduced sensitive attribute classification accuracy to 55% on the test set, demonstrating effective suppression of inference capabilities.
• The MMD-based sanitization method achieved comparable privacy protection by transforming input features to resemble either class of the sensitive attribute, effectively obscuring the original label.
• The framework maintains high utility on the primary task (e.g., diagnosis) even as it degrades performance on the protected task, validating the core trade-off between utility and privacy.
• The approach naturally promotes fairness, as models that fail to infer sensitive attributes inherently avoid discriminatory behavior based on those attributes.
• Empirical evaluation confirms that the method is robust to data shifts and can be applied to diverse real-world scenarios such as hospitals and social networks.
• The framework provides a practical, non-universal but effective solution to privacy in closed systems, where universal privacy is infeasible due to external data leakage risks.