[Paper Review] Deep Bilevel Learning

Simon Jenni, Paolo Favaro|Bern Open Repository and Information System (University of Bern)|Sep 5, 2018

Domain Adaptation and Few-Shot Learning29 references4 citations

TL;DR

This paper proposes Deep Bilevel Learning, a novel regularization method that formulates neural network training as a bilevel optimization problem to improve generalization. By dynamically adjusting mini-batch weights based on validation set performance, the method enhances generalization, especially under noisy labels, outperforming standard SGD across multiple architectures and datasets including CIFAR-10/100, ImageNet, and Pascal VOC.

ABSTRACT

We present a novel regularization approach to train neural networks that enjoys better generalization and test error than standard stochastic gradient descent. Our approach is based on the principles of cross-validation, where a validation set is used to limit the model overfitting. We formulate such principles as a bilevel optimization problem. This formulation allows us to define the optimization of a cost on the validation set subject to another optimization on the training set. The overfitting is controlled by introducing weights on each mini-batch in the training set and by choosing their values so that they minimize the error on the validation set. In practice, these weights define mini-batch learning rates in a gradient descent update equation that favor gradients with better generalization capabilities. Because of its simplicity, this approach can be integrated with other regularization methods and training schemes. We evaluate extensively our proposed algorithm on several neural network architectures and datasets, and find that it consistently improves the generalization of the model, especially when labels are noisy.

Motivation & Objective

• To address the persistent issue of overfitting in deep neural networks, especially under noisy labels, despite existing regularization techniques.
• To improve model generalization by integrating cross-validation principles directly into the training process.
• To develop a computationally efficient, plug-and-play regularization method compatible with standard training pipelines.
• To reduce the generalization gap in small-data and noisy-data scenarios through adaptive learning rate adjustments.

Proposed method

• Formulates training as a bilevel optimization problem: the upper-level optimizes validation error, while the lower-level optimizes training loss.
• Introduces scalar weights for each mini-batch to control gradient updates, favoring those that align with validation set gradients.
• Approximates the upper-level objective with quadratic functions to derive closed-form updates resembling SGD, enabling computational efficiency.
• Uses gradient agreement between training and validation sets to adjust mini-batch importance—high agreement leads to higher weights.
• Treats the mini-batch weights as hyperparameters optimized via cross-validation, enabling end-to-end joint optimization of model and regularization.
• Can be combined with other regularization techniques such as data augmentation, dropout, and label smoothing.

Experimental results

Research questions

• RQ1 Can bilevel optimization improve generalization in deep neural networks beyond standard SGD, especially when labels are noisy?
• RQ2 How does the proposed method compare to state-of-the-art regularization and noise-robust training methods on benchmark datasets?
• RQ3 To what extent can bilevel learning reduce overfitting in small-data regimes and under high label noise?
• RQ4 Does the method maintain performance gains when combined with existing regularization techniques?

Key findings

• On CIFAR-10 and CIFAR-100 with up to 50% random label noise, Deep Bilevel Learning achieves significantly higher test accuracy than standard SGD, with improvements of up to 5% on CIFAR-100 under 50% noise.
• On ImageNet with a realistic 44% label noise model derived from pre-trained network predictions, the method improves top-1 accuracy by nearly 2% (52.69% vs. 50.75%) over standard SGD.
• On the Clothing1M dataset, the method achieves 69.9% test accuracy when trained only on noisy data, comparable to state-of-the-art methods, and 79.9% after fine-tuning on clean data.
• On Pascal VOC 2007 with reduced training data, the method consistently improves mAP and reduces the generalization gap across all data fractions tested.
• When combined with data augmentation from Zhang et al. [38], the method achieves the best performance on both CIFAR-10 and CIFAR-100, outperforming prior state-of-the-art.
• The method shows robustness across diverse architectures, including ResNet, Inception, and AlexNet, and maintains gains even when combined with dropout and label smoothing.