[论文解读] More General Queries and Less Generalization Error in Adaptive Data Analysis

Raef Bassily, Adam Smith+2 more|arXiv (Cornell University)|Mar 16, 2015

Privacy-Preserving Technologies in Data参考文献 14被引用 11

一句话总结

本文通过引入更简单、更模块化的算法，利用差分隐私来降低泛化误差，改进了自适应数据分析的样本复杂度界。它首次为包括低敏感度和凸风险最小化查询在内的通用查询族提供了上界，显著超越了Dwork等人及Hardt和Ullman的先前工作。

ABSTRACT

Adaptivity is an important feature of data analysis---typically the choice of questions asked about a dataset depends on previous interactions with the same dataset. However, generalization error is typically bounded in a non-adaptive model, where all questions are specified before the dataset is drawn. Recent work by Dwork et al. (STOC '15) and Hardt and Ullman (FOCS '14) initiated the formal study of this problem, and gave the first upper and lower bounds on the achievable generalization error for adaptive data analysis. Specifically, suppose there is an unknown distribution $\mathcal{P}$ and a set of $n$ independent samples $x$ is drawn from $\mathcal{P}$. We seek an algorithm that, given $x$ as input, "accurately" answers a sequence of adaptively chosen "queries" about the unknown distribution $\mathcal{P}$. How many samples $n$ must we draw from the distribution, as a function of the type of queries, the number of queries, and the desired level of accuracy? In this work we make two new contributions towards resolving this question: *We give upper bounds on the number of samples $n$ that are needed to answer statistical queries that improve over the bounds of Dwork et al. *We prove the first upper bounds on the number of samples required to answer more general families of queries. These include arbitrary low-sensitivity queries and the important class of convex risk minimization queries. As in Dwork et al., our algorithms are based on a connection between differential privacy and generalization error, but we feel that our analysis is simpler and more modular, which may be useful for studying these questions in the future.

研究动机与目标

• 为控制自适应数据分析中的泛化误差提供解决方案，其中查询依赖于与同一数据集的先前交互。
• 改进针对统计查询的现有样本复杂度界，特别是在自适应设置下。
• 将理论保证扩展至基础统计查询之外的更通用查询族，如低敏感度和凸风险最小化查询。
• 简化并模块化基于差分隐私的泛化分析，使其对未来研究更具可及性。

提出的方法

• 作者利用差分隐私与泛化误差之间的联系，使用隐私机制来限制自适应查询工作负载中的过拟合。
• 他们设计了新的算法，在保持差分隐私的同时准确回答自适应查询，确保低泛化误差。
• 该方法具有模块化特性，可通过分析其敏感度和结构，系统地分析不同查询族。
• 对于凸风险最小化查询，该方法利用光滑性和凸性特性，推导出更紧致的样本复杂度界。
• 通过聚焦于基于敏感度的机制，避免了复杂的依赖关系，简化了先前的证明。
• 该框架支持有界和无界查询族，增强了其广泛应用的潜力。

实验结果

研究问题

• RQ1 回答自适应统计查询并保持低泛化误差所需的最少样本数是多少？
• RQ2 如何为超越基础统计查询的更复杂查询族界定泛化误差？
• RQ3 是否可以以更简单、更模块化的方式使用差分隐私，以实现更紧致的样本复杂度界？
• RQ4 在自适应设置下，低敏感度和凸风险最小化查询可实现的样本复杂度是多少？
• RQ5 与Dwork等人及Hardt和Ullman的界相比，所提出方法在样本效率方面有何改进？

主要发现

• 本文为自适应统计查询建立了改进的样本复杂度上界，优于Dwork等人先前的结果。
• 首次为低敏感度查询提供了样本复杂度的上界，扩展了可证明泛化的算法范围。
• 对于凸风险最小化查询，该方法实现了有界的泛化误差，且样本复杂度随问题参数高效缩放。
• 所提出的算法比以往方法更简单、更模块化，有利于未来扩展与分析。
• 差分隐私与泛化之间的联系以一种增强理论清晰度和实际适用性的形式被形式化。