[论文解读] Typical Stability

Raef Bassily, Yoav Freund|arXiv (Cornell University)|Apr 12, 2016

Privacy-Preserving Technologies in Data参考文献 4被引用 5

一句话总结

该论文提出了一种新型的算法稳定性概念——典型稳定性（typical stability），它在不依赖有界敏感性或独立样本的前提下，控制自适应数据分析中的泛化误差。该概念确保查询输出在其数据分布下的期望值附近集中，从而支持针对亚高斯和亚指数查询的校准噪声添加机制。

ABSTRACT

In this paper, we introduce a notion of algorithmic stability called typical stability. When our goal is to release real-valued queries (statistics) computed over a dataset, this notion does not require the queries to be of bounded sensitivity -- a condition that is generally assumed under differential privacy [DMNS06, Dwork06] when used as a notion of algorithmic stability [DFHPRR15a, DFHPRR15b, BNSSSU16] -- nor does it require the samples in the dataset to be independent -- a condition that is usually assumed when generalization-error guarantees are sought. Instead, typical stability requires the output of the query, when computed on a dataset drawn from the underlying distribution, to be concentrated around its expected value with respect to that distribution. We discuss the implications of typical stability on the generalization error (i.e., the difference between the value of the query computed on the dataset and the expected value of the query with respect to the true data distribution). We show that typical stability can control generalization error in adaptive data analysis even when the samples in the dataset are not necessarily independent and when queries to be computed are not necessarily of bounded-sensitivity as long as the results of the queries over the dataset (i.e., the computed statistics) follow a distribution with a light tail. Examples of such queries include, but not limited to, subgaussian and subexponential queries. We also discuss the composition guarantees of typical stability and prove composition theorems that characterize the degradation of the parameters of typical stability under $k$-fold adaptive composition. We also give simple noise-addition algorithms that achieve this notion. These algorithms are similar to their differentially private counterparts, however, the added noise is calibrated differently.

研究动机与目标

• 解决现有稳定性概念在自适应数据分析中对有界敏感性或独立样本的依赖所导致的局限性。
• 构建一个适用于实值查询的稳定性框架，即使样本之间存在依赖性或查询缺乏有界敏感性亦适用。
• 在关于数据依赖性和查询敏感性假设最少的前提下，提供泛化误差的理论保证。
• 建立典型稳定性在 k 重自适应组合下的组合定理。
• 设计实现典型稳定性的噪声添加机制，其噪声方差根据查询输出分布的尾部行为进行校准。

提出的方法

• 将典型稳定性定义为查询输出在其真实数据分布下的期望值附近的集中性。
• 利用亚高斯和亚指数尾部条件来刻画查询结果的集中行为。
• 制定组合定理，量化典型稳定性参数在 k 次自适应查询迭代过程中的退化情况。
• 提出噪声添加机制，其中噪声方差根据查询输出分布的尾部分布特性进行校准。
• 分析尾部衰减（轻尾分布）与泛化误差控制之间的相互作用。
• 利用针对查询输出分布量身定制的集中不等式，推导泛化误差的上界。

实验结果

研究问题

• RQ1 是否可以在不假设查询具有有界敏感性的前提下，控制自适应数据分析中的泛化误差？
• RQ2 当数据集中样本存在依赖关系时，是否仍能确保性能稳定？
• RQ3 典型稳定性在重复自适应查询下如何退化？其退化由何种组合定理所支配？
• RQ4 何种噪声校准策略可在保持效用的前提下实现典型稳定性？
• RQ5 哪些查询类别（如亚高斯、亚指数）天然满足典型稳定性？

主要发现

• 典型稳定性即使在查询不具有有界敏感性时，也能控制自适应数据分析中的泛化误差。
• 该框架适用于依赖样本，放宽了泛化误差分析中的一项标准假设。
• 组合定理表明，典型稳定性参数在 k 次自适应查询中可预测地退化。
• 实现典型稳定性的噪声添加机制类似于差分隐私机制，但其噪声校准基于查询输出的尾部分布特性。
• 亚高斯和亚指数查询因其轻尾输出分布，天然满足典型稳定性。
• 该方法在弱于差分隐私或标准泛化界所要求的假设下，实现了泛化保证。