[Paper Review] Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting

Jun Shu, Qi Xie+5 more|arXiv (Cornell University)|Feb 20, 2019

Machine Learning and Data Classification71 references398 citations

TL;DR

The paper proposes Meta-Weight-Net (MW-Net), an explicit, trainable weighting function for samples, learned via meta-learning to robustly handle biased training data (e.g., class imbalance and noisy labels) without manually designing weighting schemes.

ABSTRACT

Current deep neural networks (DNNs) can easily overfit to biased training data with corrupted labels or class imbalance. Sample re-weighting strategy is commonly used to alleviate this issue by designing a weighting function mapping from training loss to sample weight, and then iterating between weight recalculating and classifier updating. Current approaches, however, need manually pre-specify the weighting function as well as its additional hyper-parameters. It makes them fairly hard to be generally applied in practice due to the significant variation of proper weighting schemes relying on the investigated problem and training data. To address this issue, we propose a method capable of adaptively learning an explicit weighting function directly from data. The weighting function is an MLP with one hidden layer, constituting a universal approximator to almost any continuous functions, making the method able to fit a wide range of weighting functions including those assumed in conventional research. Guided by a small amount of unbiased meta-data, the parameters of the weighting function can be finely updated simultaneously with the learning process of the classifiers. Synthetic and real experiments substantiate the capability of our method for achieving proper weighting functions in class imbalance and noisy label cases, fully complying with the common settings in traditional methods, and more complicated scenarios beyond conventional cases. This naturally leads to its better accuracy than other state-of-the-art methods.

Motivation & Objective

• Motivate robust learning under biased training data (noisy labels, class imbalance).
• Eliminate manual specification of loss-to-weight mappings by learning a weighting function directly from data.
• Demonstrate that an MLP-based weight net can approximate traditional weighting schemes and adapt to complex biases.
• Provide a meta-learning framework that updates classifier and weighting function parameters jointly using a small unbiased meta-dataset.

Proposed method

• Model the sample weighting as V(L_i_train(w); Θ), an MLP with one hidden layer producing weights in [0,1].
• Optimize classifier parameters w on the weighted loss with SGD, using weights from MW-Net.
• Update MW-Net parameters Θ via a meta-loss computed on a small unbiased meta-dataset: Θ* = argmin_Θ L_meta(w*(Θ)).
• Use a bilevel/meta-learning procedure with online, alternating updates to w and Θ within a single loop (Algorithm 1).
• Provide convergence analysis showing under mild conditions the algorithm converges to critical points of the meta and training losses.

Experimental results

Research questions

• RQ1 Can an explicitly learned weighting function via MW-Net match or exceed hand-designed weighting schemes across bias settings (imbalance, noisy labels, real data)?
• RQ2 Does meta-learning MW-Net with a small unbiased meta-dataset effectively guide weighting to improve generalization?
• RQ3 Is the MW-Net approach robust to different classifier architectures and data biases?
• RQ4 What are the convergence properties of the proposed two-loop optimization in practice?
• RQ5 How interpretable are the learned weighting functions in different bias scenarios?

Key findings

• MW-Net learns weighting functions that align with traditional priors (e.g., larger weights for hard/imbalanced cases; smaller weights for noisy-label samples).
• In long-tailed CIFAR settings, MW-Net improves test accuracy over base models and several reweighting baselines, including focal loss, class-balanced, and L2RW.
• Under uniform and flip label noise, MW-Net achieves higher accuracy than many competitors across CIFAR-10/100, with notable gains at higher noise rates.
• On Clothing1M, MW-Net achieves the best accuracy among compared methods, indicating effectiveness on real-world noisy labels.
• The learned weight distributions show clean samples receiving higher weights than noisy ones, and the weight function evolves gradually and stabilizes during training.