[论文解读] Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance

Ziv Goldfeld, Kristjan Greenewald+1 more|arXiv (Cornell University)|Feb 3, 2020

Generative Adversarial Networks and Image Synthesis参考文献 83被引用 7

一句话总结

本文通过最小平滑Wasserstein估计量（MSWE）建立了生成建模的渐近保证，该方法利用平滑1-Wasserstein距离（SWD）克服维度灾难问题。证明了估计量的可测性与强一致性，并推导出在任意维度下参数估计与最小距离的$O(n^{-1/2})$收敛速率，从而实现无维度依赖的泛化界。

ABSTRACT

Minimum distance estimation (MDE) gained recent attention as a formulation of (implicit) generative modeling. It considers minimizing, over model parameters, a statistical distance between the empirical data distribution and the model. This formulation lends itself well to theoretical analysis, but typical results are hindered by the curse of dimensionality. To overcome this and devise a scalable finite-sample statistical MDE theory, we adopt the framework of smooth 1-Wasserstein distance (SWD) $\mathsf{W}_1^{(σ)}$. The SWD was recently shown to preserve the metric and topological structure of classic Wasserstein distances, while enjoying dimension-free empirical convergence rates. In this work, we conduct a thorough statistical study of the minimum smooth Wasserstein estimators (MSWEs), first proving the estimator's measurability and asymptotic consistency. We then characterize the limit distribution of the optimal model parameters and their associated minimal SWD. These results imply an $O(n^{-1/2})$ generalization bound for generative modeling based on MSWE, which holds in arbitrary dimension. Our main technical tool is a novel high-dimensional limit distribution result for empirical $\mathsf{W}_1^{(σ)}$. The characterization of a nondegenerate limit stands in sharp contrast with the classic empirical 1-Wasserstein distance, for which a similar result is known only in the one-dimensional case. The validity of our theory is supported by empirical results, posing the SWD as a potent tool for learning and inference in high dimensions.

研究动机与目标

• 为解决生成建模中最小距离估计（MDE）的维度灾难问题，该问题在标准统计距离（如Wasserstein距离与f-散度）中普遍存在。
• 为MDE建立有限样本统计理论，使其在高维设置下仍保持可扩展性与有效性。
• 为最小平滑Wasserstein估计量（MSWE）建立理论保证——包括可测性、一致性及极限分布。
• 推导基于平滑Wasserstein距离的生成建模的泛化界，其阶为$O(n^{-1/2})$，且在任意维度下均成立。

提出的方法

• 本文采用平滑1-Wasserstein距离（$\mathsf{W}_{1}^{(\sigma)}$），即数据分布与模型分布与方差为$\sigma^2$的各向同性高斯核卷积后的1-Wasserstein距离。
• 证明了MSWE $\widehat{\theta}_n = \mathop{\mathrm{argmin}}_{\theta \in \Theta} \mathsf{W}_{1}^{(\sigma)}(P_n, Q_\theta)$的可测性与强一致性。
• 推导出$\sqrt{n}\,\mathsf{W}_{1}^{(\sigma)}(P_n, P)$的新型高维极限分布结果，从而实现参数估计的渐近正态性。
• 分析基于Kantorovich-Rubinstein对偶性，将SWD表示为1-Lipschitz函数族上的经验过程上确界。
• 通过合成数据上的实证验证支持理论结果，包括SWD的收敛性以及参数估计的极限分布。

实验结果

研究问题

• RQ1 平滑1-Wasserstein距离能否为MDE提供避免维度灾难的有限样本统计理论？
• RQ2 在高维情形下，最小平滑Wasserstein估计量的渐近性质（一致性与极限分布）为何？
• RQ3 平滑Wasserstein距离是否能在任意维度下实现$O(n^{-1/2})$阶的泛化界？
• RQ4 高维情形下$\sqrt{n}\,\mathsf{W}_{1}^{(\sigma)}(P_n, P)$的极限分布与经典经验1-Wasserstein距离有何不同？

主要发现

• 最小平滑Wasserstein估计量$\widehat{\theta}_n$是可测的，且几乎必然收敛于真实参数$\theta^\star$。
• 即使在高维情形下，$\sqrt{n}(\widehat{\theta}_n - \theta^\star)$的极限分布也被明确刻画且非退化，这与经典Wasserstein距离不同。
• 最小平滑Wasserstein距离$\sqrt{n}\,\mathsf{W}_{1}^{(\sigma)}(P_n, Q_{\widehat{\theta}_n})$以$O(n^{-1/2})$的速率依分布收敛。
• 本文建立了基于MSWE的生成建模的$O(n^{-1/2})$阶泛化界，且在任意维度下均成立。
• 实证结果验证了理论收敛速率，并支持在SWD框架下参数估计的极限正态性。