[Paper Review] Making AI Forget You: Data Deletion in Machine Learning

Antonio Ginart, Melody Y. Guan+2 more|arXiv (Cornell University)|Jul 11, 2019

Privacy-Preserving Technologies in DataComputer Science85 references71 citations

TL;DR

The paper formalizes data deletion in ML and proposes two deletion-efficient k-means algorithms (Q-kmeans and DC-kmeans) with theoretical guarantees and strong empirical speedups over standard retraining, while preserving clustering quality.

ABSTRACT

Intense recent discussions have focused on how to provide individuals with control over when their data can and cannot be used --- the EU's Right To Be Forgotten regulation is an example of this effort. In this paper we initiate a framework studying what to do when it is no longer permissible to deploy models derivative from specific user data. In particular, we formulate the problem of efficiently deleting individual data points from trained machine learning models. For many standard ML models, the only way to completely remove an individual's data is to retrain the whole model from scratch on the remaining data, which is often not computationally practical. We investigate algorithmic principles that enable efficient data deletion in ML. For the specific setting of k-means clustering, we propose two provably efficient deletion algorithms which achieve an average of over 100X improvement in deletion efficiency across 6 datasets, while producing clusters of comparable statistical quality to a canonical k-means++ baseline.

Motivation & Objective

• Formalize the problem of deleting an individual data point from a trained ML model.
• Define deletion efficiency in an online/deletion setting and establish lower bounds.
• Develop deletion-efficient algorithms for k-means clustering with theoretical guarantees.
• Empirically evaluate deletion efficiency and clustering quality across diverse datasets.
• Provide a design toolbox and principles for deletion-efficient learning systems.

Proposed method

• Define data deletion as producing a model indistinguishable (in distribution) from one trained without the deleted point.
• Propose two deletion-efficient k-means variants: Quantized k-means (Q-kmeans) and Divide-and-Conquer k-means (DC-kmeans).
• Q-kmeans: quantizes centroids to an epsilon-lattice, memoizes state, applies balance correction, and uses early termination for unstable iterations.
• DC-kmeans: builds a w-ary tree, solves subproblems with k-means++, and merges centroids up the tree to enable localized deletions.
• Analyze deletion time complexity and show O(m^2 d^{5/2}/epsilon) expected time for Q-kmeans deletions under certain conditions; provide prop/proof sketches for DC-kmeans deletion bounds.
• Present four design principles (Linearity, Laziness, Modularity, Quantization) for deletion-efficient systems.

Experimental results

Research questions

• RQ1 How to define and measure data deletion for ML models in a computationally meaningful way?
• RQ2 Can we delete an individual training point efficiently without retraining from scratch, while preserving model quality?
• RQ3 What deletion-efficient strategies can be developed for k-means clustering, and what are their theoretical guarantees?
• RQ4 How do deletion-efficient methods perform empirically in terms of speed and clustering quality across diverse datasets?
• RQ5 What general engineering principles emerge for designing deletion-efficient ML systems?

Key findings

• Deletion-efficient k-means variants achieve substantial speedups (average >100x in amortized runtime) over standard Lloyd’s algorithm with k-means++ initialization.
• Q-kmeans maintains competitive clustering quality while enabling fast deletions via centroid quantization and state memoization; its deletion time is O(m^2 d^{5/2}/epsilon) in expectation.
• DC-kmeans provides another deletion strategy using a depth-1 w-ary tree that yields deletion time O(m max{n^{rho}, n^{1−rho}} d) in expectation for appropriate parameter choices.
• Across six datasets, both methods produced statistically comparable results to the baseline in key metrics (loss, silhouette, NMI) while drastically reducing deletion cost.
• The authors establish amortized deletion efficiency bounds: Q-kmeans is alpha-deletion efficient for alpha <= (1 - beta)/2 with epsilon ~ n^{-beta}; DC-kmeans is alpha-deletion efficient for alpha < 1 - max{1 - rho, rho} with tree width w ~ n^{rho}.