[Paper Review] Classification with unknown class-conditional label noise on non-compact feature spaces

Henry W. J. Reeve, Ata Kabán|arXiv (Cornell University)|Feb 14, 2019

Machine Learning and Data Classification30 references4 citations

TL;DR

This paper establishes minimax optimal learning rates for non-parametric classification under unknown class-conditional label noise in non-compact metric spaces. It shows that learning rates match the noise-free setting when the regression function approaches its extrema rapidly, but degrade when convergence is slow, and presents an adaptive algorithm achieving these optimal rates without prior knowledge of distributional parameters or local density.

ABSTRACT

We investigate the problem of classification in the presence of unknown class-conditional label noise in which the labels observed by the learner have been corrupted with some unknown class dependent probability. In order to obtain finite sample rates, previous approaches to classification with unknown class-conditional label noise have required that the regression function is close to its extrema on sets of large measure. We shall consider this problem in the setting of non-compact metric spaces, where the regression function need not attain its extrema. In this setting we determine the minimax optimal learning rates (up to logarithmic factors). The rate displays interesting threshold behaviour: When the regression function approaches its extrema at a sufficient rate, the optimal learning rates are of the same order as those obtained in the label-noise free setting. If the regression function approaches its extrema more gradually then classification performance necessarily degrades. In addition, we present an adaptive algorithm which attains these rates without prior knowledge of either the distributional parameters or the local density. This identifies for the first time a scenario in which finite sample rates are achievable in the label noise setting, but they differ from the optimal rates without label noise.

Motivation & Objective

• To determine minimax optimal learning rates for classification with unknown class-conditional label noise in non-compact metric spaces.
• To analyze how the rate at which the regression function approaches its extrema affects learning performance.
• To develop an adaptive algorithm that achieves optimal rates without prior knowledge of distributional parameters or local density.
• To identify a scenario where finite-sample rates in the label noise setting differ from those in the noise-free setting.

Proposed method

• The authors introduce a flexible tail assumption based on the decay of measure in regions where density is below a threshold, avoiding restrictive assumptions like bounded density or finite covering dimension.
• They derive minimax optimal learning rates (up to logarithmic factors) by analyzing the asymptotic behavior of the regression function in the tails of the distribution.
• An adaptive k-NN-based algorithm is proposed that selects the optimal k using a data-driven confidence interval approach, ensuring convergence without prior knowledge of the underlying distribution.
• The method relies on high-probability bounds for k-NN regression under Hölder continuity and minimal mass assumptions, using union bounds and concentration inequalities.
• The algorithm uses a confidence interval intersection strategy to select a stable k-value that balances bias and variance.
• Theoretical analysis proves that the algorithm achieves the minimax optimal rate up to logarithmic factors, even under unknown label noise.

Experimental results

Research questions

• RQ1 Under what conditions on the regression function’s tail behavior do learning rates in the label noise setting match those of the noise-free setting?
• RQ2 Can finite-sample learning rates be achieved in the presence of unknown class-conditional label noise on non-compact feature spaces?
• RQ3 Does the rate of convergence degrade when the regression function approaches its extrema more gradually in the tails?
• RQ4 Can an adaptive algorithm be designed that achieves optimal rates without prior knowledge of the distribution’s density or parameters?
• RQ5 Is there a scenario in which learning rates under label noise differ from the optimal rates in the absence of noise?

Key findings

• The minimax optimal learning rate depends critically on the rate at which the regression function approaches its extrema in the tails of the distribution.
• When the regression function approaches its extrema rapidly, the optimal learning rate matches the noise-free setting, up to logarithmic factors.
• If the regression function approaches its extrema slowly, classification performance necessarily degrades, and the learning rate becomes suboptimal.
• An adaptive algorithm is proposed that achieves the minimax optimal rate without prior knowledge of the distributional parameters or local density.
• The paper identifies the first scenario in which finite-sample rates are achievable under label noise, but differ from the optimal rates in the noise-free case.
• A simple, adaptive method for estimating the maximum of a function on a non-compact domain is introduced as a byproduct of the analysis.