[论文解读] Certifying Out-of-Domain Generalization for Blackbox Functions

Maurice Weber, Linyi Li+4 more|arXiv (Cornell University)|Feb 3, 2022

Adversarial Robustness in Machine Learning被引用 4

一句话总结

本文提出了一种可扩展的认证框架，用于黑盒机器学习模型在分布偏移下的域外泛化，利用Hellinger距离来约束最坏情况风险。该方法在ImageNet规模的数据集上，针对全精度模型（如EfficientNet-B7）和包括0-1损失与AUC在内的多种损失函数，首次实现了非平凡的认证泛化边界。

ABSTRACT

Certifying the robustness of model performance under bounded data distribution drifts has recently attracted intensive interest under the umbrella of distributional robustness. However, existing techniques either make strong assumptions on the model class and loss functions that can be certified, such as smoothness expressed via Lipschitz continuity of gradients, or require to solve complex optimization problems. As a result, the wider application of these techniques is currently limited by its scalability and flexibility -- these techniques often do not scale to large-scale datasets with modern deep neural networks or cannot handle loss functions which may be non-smooth such as the 0-1 loss. In this paper, we focus on the problem of certifying distributional robustness for blackbox models and bounded loss functions, and propose a novel certification framework based on the Hellinger distance. Our certification technique scales to ImageNet-scale datasets, complex models, and a diverse set of loss functions. We then focus on one specific application enabled by such scalability and flexibility, i.e., certifying out-of-domain generalization for large neural networks and loss functions such as accuracy and AUC. We experimentally validate our certification method on a number of datasets, ranging from ImageNet, where we provide the first non-vacuous certified out-of-domain generalization, to smaller classification tasks where we are able to compare with the state-of-the-art and show that our method performs considerably better.

研究动机与目标

• 解决黑盒模型中分布鲁棒性可扩展且灵活的认证方法缺乏的问题。
• 实现对大规模模型和非光滑损失函数（如准确率与AUC）的域外泛化认证。
• 克服先前方法依赖强假设（如Lipschitz梯度）或存在可扩展性问题的局限性。
• 提供一种实用的、与模型无关的框架，仅需域内样本，适用于任意黑盒模型和有界损失函数。

提出的方法

• 通过希尔伯特空间中的内积不等式，将最坏情况风险在Hellinger球不确定性集上进行边界约束。
• 推导出在任意与域内分布P的Hellinger距离为ρ的偏移分布Q下，期望损失的上界。
• 该边界仅依赖于P下损失的期望和方差，从而实现黑盒适用性。
• 通过Hellinger距离建模，将该方法实例化于多种域偏移类型（包括协变量偏移和标签偏移）。
• 支持广泛的有界损失函数，包括0-1损失、JSD和AUC，无需光滑性或凸性假设。
• 仅使用来自域内分布P的有限样本集进行认证，确保实际可扩展性。

实验结果

研究问题

• RQ1 我们能否在不假设模型或损失函数光滑性或Lipschitz连续性的前提下，对黑盒模型实现域外泛化的认证？
• RQ2 如何将分布鲁棒性认证扩展到ImageNet规模的数据集和现代深度神经网络？
• RQ3 与Wasserstein基方法相比，Hellinger基边界的紧致性和可扩展性如何？
• RQ4 该框架能否在大规模任务中对复杂且非光滑的损失函数（如0-1损失和AUC）提供非平凡的认证？
• RQ5 边界的紧致性如何随模型复杂度和分布偏移幅度变化？

主要发现

• 所提方法在使用EfficientNet-B7模型的ImageNet上首次实现了非平凡的认证域外泛化边界。
• 该边界保持紧致且近乎与模型无关，对模型架构复杂度的依赖性极低。
• 该方法优于先前的Wasserstein基方法，后者在模型规模增大时趋于平凡或松散。
• 首次在真实世界数据集上为包括0-1损失和AUC在内的多样化损失函数提供了认证边界。
• 实验验证了其在ImageNet上的可扩展性，并在较小基准上优于最先进方法。