[Paper Review] Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

Tang Jun, Aleksandra Korolova+3 more|arXiv (Cornell University)|Sep 8, 2017

Privacy-Preserving Technologies in DataComputer Science11 references186 citations

TL;DR

The paper analyzes Apple's local differential privacy deployment on MacOS 10.12, revealing per-datum privacy parameters but much larger total daily privacy loss due to budget management and dual-level budgeting.

ABSTRACT

In June 2016, Apple announced that it will deploy differential privacy for some user data collection in order to ensure privacy of user data, even from Apple. The details of Apple's approach remained sparse. Although several patents have since appeared hinting at the algorithms that may be used to achieve differential privacy, they did not include a precise explanation of the approach taken to privacy parameter choice. Such choice and the overall approach to privacy budget use and management are key questions for understanding the privacy protections provided by any deployment of differential privacy. In this work, through a combination of experiments, static and dynamic code analysis of macOS Sierra (Version 10.12) implementation, we shed light on the choices Apple made for privacy budget management. We discover and describe Apple's set-up for differentially private data processing, including the overall data pipeline, the parameters used for differentially private perturbation of each piece of data, and the frequency with which such data is sent to Apple's servers. We find that although Apple's deployment ensures that the (differential) privacy loss per each datum submitted to its servers is $1$ or $2$, the overall privacy loss permitted by the system is significantly higher, as high as $16$ per day for the four initially announced applications of Emojis, New words, Deeplinks and Lookup Hints. Furthermore, Apple renews the privacy budget available every day, which leads to a possible privacy loss of 16 times the number of days since user opt-in to differentially private data collection for those four applications. We advocate that in order to claim the full benefits of differentially private data collection, Apple must give full transparency of its implementation, enable user choice in areas related to privacy loss, and set meaningful defaults on the privacy loss permitted.

Motivation & Objective

• Understand how Apple implements local differential privacy in MacOS 10.12.
• Identify the privacy budget management and per-datum privacy parameters used.
• Assess how often data is reported and how privacy loss accumulates over time.
• Evaluate transparency, configurability, and potential abuse vectors in the privacy budget system.

Proposed method

• Static and dynamic code analysis of macOS Sierra (10.12) implementation.
• Decompilation and tracing of DifferentialPrivacy.framework and the dprivacyd daemon.
• Inspection of database tables, configuration files, and report files to map privacy budget and data flow.
• Experimental manipulation of configuration parameters to observe effects on privacy parameters and budget behavior.

Experimental results

Research questions

• RQ1 What per-datum privacy parameters are used for different data types before privatization?
• RQ2 How frequently are records selected for reports, and what is the maximum per-report and per-day privacy loss?
• RQ3 Is the total privacy loss per device bounded or unbounded over time?
• RQ4 How resistant is the system to parameter and timing manipulation, and what are the abuse vectors?

Key findings

• Per-datum privacy parameter values are defined per data type (e.g., emojis, new words) and align with values in the configuration files.
• The system uses a budget balance (ZBALANCE) per BudgetKeyName and a SessionAmount that increases per SessionSeconds to control budget growth.
• A report generator selects up to min(SessionAmount, 40) records per KeyName, constrained by the available budget balance, leading to per-day privacy loss equal to PrivacyParameter × SessionAmount for active types.
• Privacy budget is replenished every SessionSeconds, and unused budget rolls over, potentially yielding unbounded total privacy loss over time for four initial applications.
• Daily permitted privacy loss for the four initial apps can reach 16, and total device loss can scale with days since opt-in, due to budget replenishment mechanics.
• The implementation includes safeguards (hard-coded limits, difficulty of configuration changes) but still allows potential abuse by root access or future changes by Apple that alter budgets or parameters.
• There are differences between macOS 10.12.1 and 10.12.3 configurations, notably increased SessionAmount for NewWords and added health/local words budgets, plus SubmissionPriority.