[Paper Review] Domain Generalization without Excess Empirical Risk

Ozan Şener, Vladlen Koltun|arXiv (Cornell University)|Aug 30, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes a novel domain generalization method that minimizes a surrogate penalty only under the constraint that empirical risk remains optimal, preventing excess in-distribution error. By leveraging rate-distortion theory and a Blahut-Arimoto-style algorithm, it ensures no degradation in training performance while improving out-of-distribution generalization across multiple benchmarks and models.

ABSTRACT

Given data from diverse sets of distinct distributions, domain generalization aims to learn models that generalize to unseen distributions. A common approach is designing a data-driven surrogate penalty to capture generalization and minimize the empirical risk jointly with the penalty. We argue that a significant failure mode of this recipe is an excess risk due to an erroneous penalty or hardness in joint optimization. We present an approach that eliminates this problem. Instead of jointly minimizing empirical risk with the penalty, we minimize the penalty under the constraint of optimality of the empirical risk. This change guarantees that the domain generalization penalty cannot impair optimization of the empirical risk, i.e., in-distribution performance. To solve the proposed optimization problem, we demonstrate an exciting connection to rate-distortion theory and utilize its tools to design an efficient method. Our approach can be applied to any penalty-based domain generalization method, and we demonstrate its effectiveness by applying it to three examplar methods from the literature, showing significant improvements.

Motivation & Objective

• Address the failure mode in existing penalty-based domain generalization methods where joint optimization increases empirical risk, harming in-distribution performance.
• Identify that joint minimization of empirical risk and penalty can lead to excess risk, especially when the penalty is misaligned or hard to optimize.
• Propose a constrained optimization framework that minimizes the penalty only if it does not impair the empirical risk’s optimality.
• Ensure convergence to a stationary point of the empirical risk while minimizing the domain generalization penalty.
• Develop a general-purpose optimizer applicable to any penalty-based domain generalization method without requiring architectural changes.

Proposed method

• Formulate the domain generalization problem as minimizing the surrogate penalty under the constraint that the model remains at a stationary point of the empirical risk.
• Use biased gradient descent with bounded, decreasing bias to ensure convergence to a stationary point of the empirical risk.
• Draw a formal connection to rate-distortion theory, where the penalty acts as distortion and the empirical risk as rate.
• Apply a Blahut-Arimoto-style iterative algorithm to solve the constrained optimization problem efficiently.
• Design an update rule that minimizes the penalty while maintaining bounded deviation from the empirical gradient direction.
• Integrate the method into existing models (e.g., CORAL, FISH, VRex) with minimal modifications, preserving their original training dynamics.

Experimental results

Research questions

• RQ1 Can we eliminate excess empirical risk in domain generalization without compromising generalization performance?
• RQ2 Is it possible to decouple the optimization of empirical risk from the penalty minimization to prevent performance degradation on training distributions?
• RQ3 Can rate-distortion theory provide a principled framework for constrained domain generalization?
• RQ4 Does enforcing stationarity of the empirical risk during penalty minimization lead to consistent performance gains across diverse benchmarks?
• RQ5 How effective is the proposed method when applied to multiple existing domain generalization models?

Key findings

• The proposed method significantly improves out-of-distribution performance on the WILDS benchmark across image, natural language, formal language, and graph modalities.
• On the WILDS benchmark, the method outperforms standard ERM and existing domain generalization methods like CORAL, FISH, and VRex, even though ERM often performs better in practice.
• The method achieves consistent improvements when applied to three distinct domain generalization models: CORAL, FISH, and VRex, demonstrating broad applicability.
• Empirical results show that the method reduces the generalization gap while maintaining or improving in-distribution accuracy, validating the hypothesis that excess empirical risk harms performance.
• The Blahut-Arimoto-based optimization converges reliably and enables effective penalty minimization without degrading empirical risk optimization.
• The method introduces additional computational cost due to per-domain gradient computation, but the performance gains justify the overhead in large-scale settings.