[Paper Review] Robust Inference via Generative Classifiers for Handling Noisy Labels

Kimin Lee, Sukmin Yun+4 more|arXiv (Cornell University)|Jan 31, 2019

Machine Learning and Data ClassificationComputer Science61 citations

TL;DR

RoG builds a robust generative classifier on top of pre-trained discriminative models using minimum covariance determinant (MCD) to handle noisy labels, improving accuracy without retraining the original network, and can be ensembleed across layers.

ABSTRACT

Large-scale datasets may contain significant proportions of noisy (incorrect) class labels, and it is well-known that modern deep neural networks (DNNs) poorly generalize from such noisy training datasets. To mitigate the issue, we propose a novel inference method, termed Robust Generative classifier (RoG), applicable to any discriminative (e.g., softmax) neural classifier pre-trained on noisy datasets. In particular, we induce a generative classifier on top of hidden feature spaces of the pre-trained DNNs, for obtaining a more robust decision boundary. By estimating the parameters of generative classifier using the minimum covariance determinant estimator, we significantly improve the classification accuracy with neither re-training of the deep model nor changing its architectures. With the assumption of Gaussian distribution for features, we prove that RoG generalizes better than baselines under noisy labels. Finally, we propose the ensemble version of RoG to improve its performance by investigating the layer-wise characteristics of DNNs. Our extensive experimental results demonstrate the superiority of RoG given different learning models optimized by several training techniques to handle diverse scenarios of noisy labels.

Motivation & Objective

• Motivate robust generalization when training data has noisy labels.
• Propose RoG to post-process pre-trained discriminative models with a generative classifier.
• Show that MCD-based estimation improves robustness and generalization under noise.
• Demonstrate ensemble RoG across multiple layers to further enhance performance.
• Demonstrate compatibility with various training techniques for noisy labels.

Proposed method

• Induce a generative classifier on top of the pre-trained penultimate feature f(x) assuming class-conditional Gaussians with a tied covariance (LDA).
• Estimate class means mu_c and shared covariance Sigma using robust minimum covariance determinant (MCD) instead of naive sample estimates.
• Compute posteriors P(y=c|f(x)) via Bayes with P(f(x)|y=c)=N(f(x)|mu_c, Sigma) and P(y=c) prior beta_c.
• Provide an approximate MCD solver (Algorithm 1) and combine class-wise covariances into a global tied Sigma.
• Propose an ensemble variant that aggregates RoG posteriors across low-level features and layers with weights learned by validation NLL.
• Demonstrate RoG compatibility as an inference method that can accompany existing noisy-label training strategies.

Experimental results

Research questions

• RQ1 Can a robust generative classifier applied at inference on top of a pre-trained discriminative model improve performance under noisy labels without retraining?
• RQ2 Does the MCD-based parameter estimation yield better generalization and robustness than naive sample estimates under label noise?
• RQ3 Does ensembling RoG across layers/feature spaces provide additional performance gains?
• RQ4 How does RoG interact with existing noisy-label training methods and semi-clean data settings?
• RQ5 Is RoG effective across vision and NLP tasks with various noise types (uniform, flip, semantic, open-set)?

Key findings

• RoG with MCD estimation improves test accuracy over softmax when training data has uniform, flip, semantic, or open-set noise across CIFAR-10/100, SVHN, and NLP tasks.
• Under CIFAR-10 with uniform noise, RoG with MCD achieves higher accuracy than softmax and non-MCD generative classifiers, and gains are amplified by ensemble across layers.
• Ensembling RoG across low-level and high-level features yields further improvements, especially since low-level RoG is more robust to high noise fractions.
• RoG consistently improves performance when combined with state-of-the-art noisy-label training methods (e.g., Han et al. 2018b) and Co-teaching, often achieving the best results in tested settings.
• Open-set and semantically noisy labels see RoG providing comparable or superior performance to softmax, with notable gains in many configurations.