[Paper Review] What is the Effect of Importance Weighting in Deep Learning?

Jonathon Byrd, Zachary C. Lipton|arXiv (Cornell University)|Dec 8, 2018

Domain Adaptation and Few-Shot Learning30 references113 citations

TL;DR

The paper shows that importance weighting significantly affects deep networks early in training, but its impact fades with further training; regularization and batch norm can partially restore it, while dropout does not.

ABSTRACT

Importance-weighted risk minimization is a key ingredient in many machine learning algorithms for causal inference, domain adaptation, class imbalance, and off-policy reinforcement learning. While the effect of importance weighting is well-characterized for low-capacity misspecified models, little is known about how it impacts over-parameterized, deep neural networks. This work is inspired by recent theoretical results showing that on (linearly) separable data, deep linear networks optimized by SGD learn weight-agnostic solutions, prompting us to ask, for realistic deep networks, for which many practical datasets are separable, what is the effect of importance weighting? We present the surprising finding that while importance weighting impacts models early in training, its effect diminishes over successive epochs. Moreover, while L2 regularization and batch normalization (but not dropout), restore some of the impact of importance weighting, they express the effect via (seemingly) the wrong abstraction: why should practitioners tweak the L2 regularization, and by how much, to produce the correct weighting effect? Our experiments confirm these findings across a range of architectures and datasets.

Motivation & Objective

• Investigate how importance-weighted risk minimization (IW-ERM) affects deep neural networks in realistic, over-parameterized settings.
• Examine the roles of early stopping, regularization (L2, dropout), and batch normalization in modulating IW-ERM effects.
• Assess whether IW-ERM’s impact persists across architectures, datasets, and tasks (images and text).
• Provide practical guidance for using importance weighting in domains like causal inference, domain adaptation, and off-policy learning.

Proposed method

• Ground the study in theoretical results on weight direction vs. magnitude from Soudry et al. and Gunasekar et al. to motivate weight-agnostic behavior on separable data.
• Conduct extensive experiments on synthetic 2D datasets to visualize decision boundaries under varying weights.
• Evaluate IW-ERM on CIFAR-10 with binary and multiclass setups using CNNs and ResNet variants, with and without batch normalization, L2, or dropout.
• Test on natural language with MRPC by fine-tuning BERT-like models under different importance weights.
• Compare SGD and Adam optimizers to assess robustness of findings across optimization methods.

Experimental results

Research questions

• RQ1 Does importance weighting alter the decision boundaries of modern deep networks when trained with SGD on separable data?
• RQ2 How do regularization (L2, dropout) and batch normalization interact with IW-ERM in deep networks?
• RQ3 Does the impact of importance weighting persist as training epochs increase across architectures and datasets?
• RQ4 Can importance weighting meaningfully correct for label shift in deep models, and under what training conditions?
• RQ5 Are the findings consistent across architectures (MLP, CNN, ResNet, transformer-based models) and domains (synthetic data, image, text)?

Key findings

• Importance weighting significantly changes early training outcomes, but its effect diminishes with successive training epochs.
• L2 regularization and batch normalization can restore some weighting effects, while dropout does not show the same interaction.
• Across architectures, tasks, and datasets, models converge to similar solutions despite different weighting schemes as training progresses.
• In CIFAR-10, IW-ERM provides a benefit early on for label shift correction that fades with more epochs.
• Even with strong weighting, models trained on CIFAR-10 and MRPC show high agreement on test labels across different weightings, indicating similar decision boundaries.
• Weighting can slow convergence and may require many epochs (far beyond typical fine-tuning) to stabilize effects in some networks.