[Paper Review] Instance-Dependent Noisy Label Learning via Graphical Modelling

Arpit Garg, Cuong C. Nguyen+3 more|arXiv (Cornell University)|Sep 2, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes InstanceGM, a graphical modelling approach for instance-dependent noisy label learning that combines a variational auto-encoder with a continuous Bernoulli distribution for image reconstruction and uses DivideMix as a discriminative classifier to leverage both clean and noisy labels. The method achieves state-of-the-art accuracy on multiple IDN benchmarks, including 95.90% on CIFAR10 at 50% noise rate, demonstrating significant improvements over prior methods through better training stability and full-data utilization.

ABSTRACT

Noisy labels are unavoidable yet troublesome in the ecosystem of deep learning because models can easily overfit them. There are many types of label noise, such as symmetric, asymmetric and instance-dependent noise (IDN), with IDN being the only type that depends on image information. Such dependence on image information makes IDN a critical type of label noise to study, given that labelling mistakes are caused in large part by insufficient or ambiguous information about the visual classes present in images. Aiming to provide an effective technique to address IDN, we present a new graphical modelling approach called InstanceGM, that combines discriminative and generative models. The main contributions of InstanceGM are: i) the use of the continuous Bernoulli distribution to train the generative model, offering significant training advantages, and ii) the exploration of a state-of-the-art noisy-label discriminative classifier to generate clean labels from instance-dependent noisy-label samples. InstanceGM is competitive with current noisy-label learning approaches, particularly in IDN benchmarks using synthetic and real-world datasets, where our method shows better accuracy than the competitors in most experiments.

Motivation & Objective

• Address the challenge of instance-dependent label noise (IDN), where mislabelling depends on image content, which is common in real-world datasets but underexplored in prior methods.
• Overcome limitations of existing approaches that rely on instance-independent noise assumptions or use only clean samples in training.
• Develop a unified graphical model framework that treats the true label and image features as latent variables to jointly model image and noisy label generation.
• Improve training stability and performance by replacing mean squared error (MSE) with the continuous Bernoulli distribution for image reconstruction.
• Enable full utilization of training data by replacing co-teaching with DivideMix, which incorporates both clean and noisy samples in semi-supervised learning.

Proposed method

• InstanceGM employs a graphical model where the true label $Y$ and image feature $Z$ are latent variables, and the observed data are the image $X$ and its noisy label $\hat{Y}$.
• The generative model uses a variational auto-encoder (VAE) with a continuous Bernoulli distribution to model the likelihood of reconstructed images, replacing standard MSE loss to improve training and eliminate hyperparameter tuning.
• The continuous Bernoulli distribution is used to model pixel-level reconstruction, providing better gradient flow and stability during optimization.
• The discriminative component uses DivideMix, a semi-supervised learning method that trains on both clean and noisy labels by leveraging pseudo-labeling and consistency regularization.
• The framework jointly optimizes the generative and discriminative components through an end-to-end training procedure, enabling robust learning under high IDN rates.
• The method is evaluated on synthetic and real-world IDN benchmarks, including CIFAR10, CIFAR100, Red Mini-ImageNet, ANIMAL-10N, and CLOTHING-1M, with consistent performance gains.

Figure 2: The proposed InstanceGM trains the Classifiers to output clean labels for instance-dependent noisy-label samples. We first warmup our two classifiers (Classifier- { 11,12 } ) using the classification loss, and then with classification loss we train the GMM to separate clean and noisy sampl

Experimental results

Research questions

• RQ1 Can a graphical model that jointly models image and noisy label generation improve performance on instance-dependent noisy label learning compared to existing methods?
• RQ2 Does replacing the standard MSE reconstruction loss with the continuous Bernoulli distribution lead to more stable and accurate training in VAE-based generative models for noisy label learning?
• RQ3 Can the use of DivideMix as the discriminative classifier, which leverages both clean and noisy samples, outperform co-teaching-based methods that only use clean samples?
• RQ4 How does InstanceGM perform across diverse IDN benchmarks, including high-noise synthetic datasets and real-world noisy datasets like CLOTHING-1M and ANIMAL-10N?
• RQ5 What is the contribution of each component—continuous Bernoulli reconstruction and DivideMix—to the overall performance gain over baseline methods like CausalNL?

Key findings

• InstanceGM achieves 95.90% test accuracy on CIFAR10 under 50% instance-dependent noise, significantly outperforming CausalNL (78.63%) and CausalNL with DivideMix (88.62%).
• On ANIMAL-10N, InstanceGM achieves 84.6% accuracy using a VGG backbone, with further gains using ResNet (82.2%) and ConvNeXt (84.7%), demonstrating strong generalization across architectures.
• On Red Mini-ImageNet, InstanceGM achieves state-of-the-art performance both without and with self-supervised pre-training, outperforming PropMix and other recent methods.
• In the ablation study, replacing co-teaching with DivideMix improves accuracy by approximately 10%, and switching from MSE to continuous Bernoulli reconstruction further boosts performance by around 7%.
• On CLOTHING-1M, InstanceGM achieves 74.40% accuracy, ranking competitively among state-of-the-art methods, including DivideMix (74.76%) and Nested-CoTeaching (74.90%), demonstrating robustness on large-scale real-world noisy datasets.
• The ablation study confirms that both the continuous Bernoulli reconstruction loss and the DivideMix-based discriminative classifier are critical to InstanceGM’s success, with each component contributing substantially to the final performance gain.