[Paper Review] On the Resistance of Nearest Neighbor to Random Noisy Labels

Wei Gao, Binbin Yang+1 more|arXiv (Cornell University)|Jul 26, 2016

Machine Learning and Data Classification66 references15 citations

TL;DR

This paper proposes Robust $k$-Nearest Neighbor (R$k$NN), a method that leverages the inherent robustness of $k$-NN to random noisy labels by correcting only the most severely misled examples. Theoretically, it proves $k$-NN is consistent under symmetric noise and robust under asymmetric noise, with empirical validation showing strong performance across benchmark datasets even with imprecise noise estimation.

ABSTRACT

Nearest neighbor has always been one of the most appealing non-parametric approaches in machine learning, pattern recognition, computer vision, etc. Previous empirical studies partly shows that nearest neighbor is resistant to noise, yet there is a lack of deep analysis. This work presents the finite-sample and distribution-dependent bounds on the consistency of nearest neighbor in the random noise setting. The theoretical results show that, for asymmetric noises, k-nearest neighbor is robust enough to classify most data correctly, except for a handful of examples, whose labels are totally misled by random noises. For symmetric noises, however, k-nearest neighbor achieves the same consistent rate as that of noise-free setting, which verifies the resistance of k-nearest neighbor to random noisy labels. Motivated by the theoretical analysis, we propose the Robust k-Nearest Neighbor (RkNN) approach to deal with noisy labels. The basic idea is to make unilateral corrections to examples, whose labels are totally misled by random noises, and classify the others directly by utilizing the robustness of k-nearest neighbor. We verify the effectiveness of the proposed algorithm both theoretically and empirically.

Motivation & Objective

• To theoretically analyze the consistency of $k$-NN under random label noise, especially in finite-sample and distribution-dependent settings.
• To understand why $k$-NN is empirically observed to be robust to noisy labels, despite limited theoretical justification.
• To develop a practical algorithm that exploits this robustness while correcting only the most corrupted examples.
• To validate the theoretical findings empirically on synthetic and real-world datasets with varying noise levels.

Proposed method

• Proposes R$k$NN, which identifies and corrects only examples whose labels are completely misled by random noise.
• Uses $k'$-nearest neighbors to estimate noise proportions $\hat{\tau}_+$ and $\hat{\tau}_-$, guiding correction decisions.
• Applies unilateral label correction only to instances with high confidence of being mislabeled, preserving $k$-NN's robustness for the rest.
• Employs a Bayesian framework to estimate noise rates and guide parameter selection, ensuring theoretical consistency.
• Integrates nearest neighbor estimation to detect and correct mislabeled instances without relying on parametric assumptions.
• Employs Bayesian t-test to statistically validate performance gains over baselines across multiple datasets.

Experimental results

Research questions

• RQ1 How does $k$-NN perform in terms of consistency under random label noise, particularly in finite-sample and distribution-dependent settings?
• RQ2 Why is $k$-NN robust to random label noise, and under what conditions does this robustness break down?
• RQ3 Can we design a practical algorithm that leverages $k$-NN's robustness while correcting only the most severely corrupted examples?
• RQ4 How accurate is the noise estimation via $k'$-NN in real-world datasets, and how sensitive is R$k$NN to parameter choices?
• RQ5 Does R$k$NN outperform existing methods in terms of classification accuracy under various noise settings?

Key findings

• For symmetric label noise, $k$-NN achieves the same consistent rate as in the noise-free setting, confirming its resistance to random noise.
• For asymmetric noise, $k$-NN remains consistent and classifies most examples correctly, except for a small number of instances whose labels are entirely misled.
• Theoretical analysis proves that $1$-NN is inconsistent even under symmetric noise, highlighting the importance of $k > 1$.
• R$k$NN achieves state-of-the-art performance on benchmark datasets, winning 53 out of 64 comparisons in Bayesian t-test.
• Noise estimation via $k'$-NN is rough but sufficient for R$k$NN to maintain high performance, demonstrating robustness to estimation errors.
• Empirical results show R$k$NN is stable and convergent with increasing sample size, and insensitive to $k$ as long as $k \geq 10$.