[Paper Review] Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning

Yu Yao, Tongliang Liu+5 more|arXiv (Cornell University)|Jun 14, 2020

Machine Learning and Data Classification49 references105 citations

TL;DR

The paper introduces a dual- T estimator that factorizes the transition matrix into two easier-to-estimate matrices via an intermediate class, reducing estimation error and improving classification under label noise.

ABSTRACT

The transition matrix, denoting the transition relationship from clean labels to noisy labels, is essential to build statistically consistent classifiers in label-noise learning. Existing methods for estimating the transition matrix rely heavily on estimating the noisy class posterior. However, the estimation error for noisy class posterior could be large due to the randomness of label noise, which would lead the transition matrix to be poorly estimated. Therefore, in this paper, we aim to solve this problem by exploiting the divide-and-conquer paradigm. Specifically, we introduce an intermediate class to avoid directly estimating the noisy class posterior. By this intermediate class, the original transition matrix can then be factorized into the product of two easy-to-estimate transition matrices. We term the proposed method the dual-T estimator. Both theoretical analyses and empirical results illustrate the effectiveness of the dual-T estimator for estimating transition matrices, leading to better classification performances.

Motivation & Objective

• Motivate reducing estimation error in the transition matrix for label-noise learning.
• Introduce a divide-and-conquer framework with an intermediate class to factorize the transition matrix.
• Propose practical estimators for the two new transition matrices and prove their theoretical advantages.
• Demonstrate empirical gains in transition-matrix estimation and downstream classification across datasets.

Proposed method

• Represent the original transition matrix T as T = T^{3spadesuit} T^{3clubsuit} using an intermediate class Y' (Eq. 2).
• Define T^{3clubsuit}_{il} = P(Y'=l|Y=i) and estimate it with the same T-estimation approach used for T, leveraging anchor points or estimated noisy posteriors.
• Define T^{3spadesuit}_{lj}(Y=i) = P(ar{Y}=j|Y'=l, Y=i) and estimate it by counting after generating intermediate labels (Eq. 4).
• Set P(Y'|x) = romP(ar{Y}|x) to connect the intermediate class to the noisy labels.
• Provide an algorithm (Algorithm 1) to compute orInt and orNoisy via noisy data and anchor-point estimation.
• Argue that the two new matrices are easier to estimate than T, with zero estimation error for T^{3clubsuit} under anchor points and reduced error for T^{3spadesuit} by using noisy-label predictions.

Experimental results

Research questions

• RQ1 Can a divide-and-conquer strategy improve the identifiability and estimation accuracy of the transition matrix in label-noise learning?
• RQ2 Does introducing an intermediate class Y' and factorizing T into T^{3spadesuit} and T^{3clubsuit} reduce estimation error compared to directly estimating T?
• RQ3 What theoretical guarantees support the claimed reduction in estimation error for the dual- T estimator?
• RQ4 How does the dual- T estimator affect downstream classifier performance across synthetic and real-world noisy-label datasets?

Key findings

• The dual- T estimator reduces estimation error for the transition matrix compared to the standard T estimator.
• The estimation error for T^{3clubsuit} is essentially zero under anchor points, making it easier to estimate.
• The estimation error for T^{3spadesuit} relies on predicting noisy labels, which is easier than estimating noisy posteriors, leading to overall lower error.
• Empirical results on MNIST, Fashion-MNIST, CIFAR10, CIFAR100, and Clothing1M show the dual- T estimator outperforms the T estimator in most large-sample settings, enhancing classification accuracy across several baselines.
• Theoretical analysis (Theorem 1) corroborates that the dual- T estimator has smaller estimation error under reasonable assumptions.