[Paper Review] Is Importance Weighting Incompatible with Interpolating Classifiers?

Ke Alexander Wang, Niladri S. Chatterji+2 more|arXiv (Cornell University)|Dec 24, 2021

Neural Networks and Applications4 citations

TL;DR

This paper resolves the paradox of importance weighting's ineffectiveness in overparameterized models by showing that exponentially-tailed losses like cross-entropy suppress the influence of importance weights. The authors propose polynomially-tailed losses that restore the corrective power of importance weighting, demonstrating theoretically and empirically that exponentiated importance weights with such losses improve generalization under label and subpopulation shift, achieving up to 9% higher test accuracy than reweighted cross-entropy.

ABSTRACT

Importance weighting is a classic technique to handle distribution shifts. However, prior work has presented strong empirical and theoretical evidence demonstrating that importance weights can have little to no effect on overparameterized neural networks. Is importance weighting truly incompatible with the training of overparameterized neural networks? Our paper answers this in the negative. We show that importance weighting fails not because of the overparameterization, but instead, as a result of using exponentially-tailed losses like the logistic or cross-entropy loss. As a remedy, we show that polynomially-tailed losses restore the effects of importance reweighting in correcting distribution shift in overparameterized models. We characterize the behavior of gradient descent on importance weighted polynomially-tailed losses with overparameterized linear models, and theoretically demonstrate the advantage of using polynomially-tailed losses in a label shift setting. Surprisingly, our theory shows that using weights that are obtained by exponentiating the classical unbiased importance weights can improve performance. Finally, we demonstrate the practical value of our analysis with neural network experiments on a subpopulation shift and a label shift dataset. When reweighted, our loss function can outperform reweighted cross-entropy by as much as 9% in test accuracy. Our loss function also gives test accuracies comparable to, or even exceeding, well-tuned state-of-the-art methods for correcting distribution shifts.

Motivation & Objective

• To resolve the paradox of why importance weighting fails in overparameterized interpolating classifiers despite being theoretically sound.
• To identify the root cause of this failure, which the authors show is the use of exponentially-tailed losses like cross-entropy, not overparameterization itself.
• To propose a new training objective using polynomially-tailed losses that preserve the influence of importance weights under gradient descent.
• To theoretically and empirically validate that exponentiated importance weights with polynomially-tailed losses improve generalization in label and subpopulation shift settings.
• To demonstrate practical superiority over reweighted cross-entropy and state-of-the-art methods in correcting distribution shift on neural networks.

Proposed method

• Propose a new loss function with polynomially decaying tails instead of exponentially decaying ones, such as the logistic or cross-entropy loss.
• Use exponentiated importance weights, where weights are raised to a power $ c $, with $ c > 1 $, to amplify the effect of rare but important examples.
• Theoretical analysis shows that gradient descent on polynomially-tailed losses converges to a direction that depends on both data and importance weights, unlike the maximum-margin classifier.
• Normalize importance weights across the full training set to stabilize training, avoiding instability from minibatch-level normalization.
• Apply the method to neural networks using early stopping based on importance-weighted validation accuracy to prevent overfitting.
• Use unbiased importance weights computed from group frequencies in training and test sets, with exponentiation and normalization to ensure stable optimization.

Experimental results

Research questions

• RQ1 Why does importance weighting fail in overparameterized models trained with exponentially-tailed losses like cross-entropy?
• RQ2 Can importance weighting be restored in interpolating models by modifying the loss function structure?
• RQ3 Does using exponentiated importance weights with polynomially-tailed losses improve generalization under label shift?
• RQ4 What is the implicit bias of gradient descent when minimizing reweighted polynomially-tailed losses in linear models?
• RQ5 Can this approach outperform reweighted cross-entropy and state-of-the-art methods in correcting distribution shift on neural networks?

Key findings

• The failure of importance weighting in overparameterized models is due to exponentially-tailed losses, not overparameterization per se.
• Polynomially-tailed losses restore the influence of importance weights, enabling gradient descent to converge to a direction dependent on both data and weights.
• With exponentiated importance weights, the test error of the classifier decays to zero in the large sample limit under label shift, unlike the maximum margin classifier which has a lower bound of 1/8 on test error.
• On the imbalanced binary CIFAR10 dataset, the proposed method achieves 67.3% mean test accuracy, outperforming reweighted cross-entropy (57.8%) and matching or exceeding state-of-the-art methods.
• On the subsampled CelebA dataset with spurious correlations, the method achieves 82.4% mean test accuracy and 53.3% worst-group accuracy, outperforming reweighted cross-entropy (79.3% and 43.1%) and competing with or surpassing DRO-based methods.
• In some settings, the proposed method improves test accuracy by up to 9% compared to reweighted cross-entropy, demonstrating strong practical gains.