[Paper Review] Certifying Out-of-Domain Generalization for Blackbox Functions

Maurice Weber, Linyi Li+4 more|arXiv (Cornell University)|Feb 3, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes a scalable certification framework for out-of-domain generalization in blackbox machine learning models using Hellinger distance to bound worst-case risk under distribution shifts. It achieves the first non-vacuous certified generalization bounds on ImageNet-scale datasets with full-precision models like EfficientNet-B7 and diverse loss functions including 0-1 and AUC.

ABSTRACT

Certifying the robustness of model performance under bounded data distribution drifts has recently attracted intensive interest under the umbrella of distributional robustness. However, existing techniques either make strong assumptions on the model class and loss functions that can be certified, such as smoothness expressed via Lipschitz continuity of gradients, or require to solve complex optimization problems. As a result, the wider application of these techniques is currently limited by its scalability and flexibility -- these techniques often do not scale to large-scale datasets with modern deep neural networks or cannot handle loss functions which may be non-smooth such as the 0-1 loss. In this paper, we focus on the problem of certifying distributional robustness for blackbox models and bounded loss functions, and propose a novel certification framework based on the Hellinger distance. Our certification technique scales to ImageNet-scale datasets, complex models, and a diverse set of loss functions. We then focus on one specific application enabled by such scalability and flexibility, i.e., certifying out-of-domain generalization for large neural networks and loss functions such as accuracy and AUC. We experimentally validate our certification method on a number of datasets, ranging from ImageNet, where we provide the first non-vacuous certified out-of-domain generalization, to smaller classification tasks where we are able to compare with the state-of-the-art and show that our method performs considerably better.

Motivation & Objective

• To address the lack of scalable and flexible certification methods for distributional robustness in blackbox models.
• To enable certification of out-of-domain generalization for large-scale models and non-smooth loss functions such as accuracy and AUC.
• To overcome limitations of prior methods that rely on strong assumptions like Lipschitz gradients or suffer from scalability issues.
• To provide a practical, model-agnostic framework that only requires in-domain samples and works with arbitrary blackbox models and bounded loss functions.

Proposed method

• The framework bounds the worst-case risk over a Hellinger ball uncertainty set using inner product inequalities in a Hilbert space.
• It derives an upper bound on the expected loss under any shifted distribution Q within a Hellinger distance ρ from the in-domain distribution P.
• The bound depends only on the expected loss and variance of the loss under P, enabling blackbox applicability.
• The method is instantiated for different domain shift types, including covariate and label shifting, via Hellinger distance modeling.
• It supports a wide range of bounded loss functions, including 0-1 loss, JSD, and AUC, without requiring smoothness or convexity.
• Certification is performed using only a finite sample set from the in-domain distribution P, ensuring practical scalability.

Experimental results

Research questions

• RQ1 Can we certify out-of-domain generalization for blackbox models without assuming smoothness or Lipschitz continuity of the model or loss?
• RQ2 How can we scale distributional robustness certification to ImageNet-scale datasets and modern deep neural networks?
• RQ3 What is the performance of Hellinger-based bounds compared to Wasserstein-based methods in terms of tightness and scalability?
• RQ4 Can the framework provide non-vacuous certificates for complex, non-smooth loss functions like 0-1 loss and AUC on large-scale tasks?
• RQ5 How does the bound's tightness vary with model complexity and distribution shift magnitude?

Key findings

• The proposed method achieves the first non-vacuous certified out-of-domain generalization bounds on ImageNet with an EfficientNet-B7 model.
• The bound remains tight and nearly model-agnostic, showing minimal dependence on model architecture complexity.
• The method outperforms prior Wasserstein-based approaches, which become vacuous or loose as model size increases.
• It provides the first certified bounds for diverse loss functions including 0-1 loss and AUC on real-world datasets.
• Experiments confirm scalability to ImageNet and superior performance on smaller benchmarks compared to state-of-the-art.