[论文解读] Minimax experimental design: Bridging the gap between statistical and worst-case approaches to least squares regression

Michał Dereziński, Kenneth L. Clarkson+2 more|arXiv (Cornell University)|Feb 4, 2019

Gaussian Processes and Bayesian Inference参考文献 24被引用 8

一句话总结

本文提出q-重缩放体积采样，一种新颖的实验设计方法，将最小二乘回归中的统计方法与最坏情况方法相连接。通过利用数据点上的分布q，并确保k ≥ d个样本，该方法以高概率实现了最优极小化最大风险边界，提供了一个统一的框架，优于经典设计与最坏情况分析。

ABSTRACT

In experimental design, we are given a large collection of vectors, each with a hidden response value that we assume derives from an underlying linear model, and we wish to pick a small subset of the vectors such that querying the corresponding responses will lead to a good estimator of the model. A classical approach in statistics is to assume the responses are linear, plus zero-mean i.i.d. Gaussian noise, in which case the goal is to provide an unbiased estimator with smallest mean squared error (A-optimal design). A related approach, more common in computer science, is to assume the responses are arbitrary but fixed, in which case the goal is to estimate the least squares solution using few responses, as quickly as possible, for worst-case inputs. Despite many attempts, characterizing the relationship between these two approaches has proven elusive. We address this by proposing a framework for experimental design where the responses are produced by an arbitrary unknown distribution. We show that there is an efficient randomized experimental design procedure that achieves strong variance bounds for an unbiased estimator using few responses in this general model. Nearly tight bounds for the classical A-optimality criterion, as well as improved bounds for worst-case responses, emerge as special cases of this result. In the process, we develop a new algorithm for a joint sampling distribution called volume sampling, and we propose a new i.i.d. importance sampling method: inverse score sampling. A key novelty of our analysis is in developing new expected error bounds for worst-case regression by controlling the tail behavior of i.i.d. sampling via the jointness of volume sampling. Our result motivates a new minimax-optimality criterion for experimental design which can be viewed as an extension of both A-optimal design and sampling for worst-case regression.

研究动机与目标

• 弥合最小二乘回归实验设计中统计方法与最坏情况方法之间的差距。
• 开发一种采样方法，使其在概率与对抗性设置下均保持强大的理论保证。
• 在对数据与设计分布的最小假设下，确保最优极小化最大风险边界。
• 通过在数据点上引入分布q，对体积采样进行推广，实现灵活且稳健的设计。
• 提供一个统一框架，在期望与高概率情形下均实现最优性能。

提出的方法

• 提出q-重缩放体积采样，即索引序列π ∈ [n]^k上的概率分布，其中每个序列被选中的概率与det(XᵀSπᵀSπX)成正比。
• 将采样概率定义为Pr(π) = [det(XᵀSπᵀSπX) / det(XᵀX)] × [d! / (k choose d)] × ∏_{i=1}^k q_i，以确保正确归一化。
• 通过q的重缩放，平衡设计选择中的统计效率与最坏情况鲁棒性。
• 将该方法应用于k ≥ d个样本，保证满秩估计并最小化最坏情况预测误差。
• 推导出期望风险与高概率风险的理论边界，表明其在极小化最大意义下具有最优性。
• 证明该方法即使在对抗性设置下，也能以常数因子内达到极小化最大风险。

实验结果

研究问题

• RQ1 是否存在一种单一的实验设计方法，可在最小二乘回归中同时实现统计假设与最坏情况假设下的最优性能？
• RQ2 如何将体积采样推广以引入对数据点的先验分布q，以提升鲁棒性？
• RQ3 所提出的q-重缩放体积采样方法的极小化最大风险是多少？与现有方法相比如何？
• RQ4 该方法是否在高概率下保持最优风险边界，而不仅是在期望下？
• RQ5 该框架能否以合理方式统一经典统计设计与最坏情况分析？

主要发现

• 所提出的q-重缩放体积采样在最小二乘回归中，以常数因子内达到极小化最大风险。
• 在相同假设下，该方法以高概率保证最优性能，而不仅是在期望下。
• 采样分布通过基于行列式的概率规则并以q_i缩放而显式定义，确保理论可处理性。
• 该框架统一了统计与最坏情况方法，提供一种在两种情形下均表现良好的单一方法。
• 该结果在k ≥ d条件下成立，确保满秩估计与设计的稳定性。
• 该方法通过引入分布q改进了先前的体积采样，实现了更好的自适应性与鲁棒性。