[Paper Review] Distance and Similarity Measures Effect on the Performance of K-Nearest Neighbor Classifier -- A Review

V. B. Surya Prasath, Haneen Arafat Abu Alfeilat+5 more|arXiv (Cornell University)|Aug 14, 2017

Machine Learning and Data Classification72 references97 citations

TL;DR

This review evaluates a large set of distance/similarity measures (up to 54) for KNN across many real-world datasets, including noise, to identify which metrics yield the best accuracy and robustness.

ABSTRACT

The K-nearest neighbor (KNN) classifier is one of the simplest and most common classifiers, yet its performance competes with the most complex classifiers in the literature. The core of this classifier depends mainly on measuring the distance or similarity between the tested examples and the training examples. This raises a major question about which distance measures to be used for the KNN classifier among a large number of distance and similarity measures available? This review attempts to answer this question through evaluating the performance (measured by accuracy, precision and recall) of the KNN using a large number of distance measures, tested on a number of real-world datasets, with and without adding different levels of noise. The experimental results show that the performance of KNN classifier depends significantly on the distance used, and the results showed large gaps between the performances of different distances. We found that a recently proposed non-convex distance performed the best when applied on most datasets comparing to the other tested distances. In addition, the performance of the KNN with this top performing distance degraded only about $20\%$ while the noise level reaches $90\%$, this is true for most of the distances used as well. This means that the KNN classifier using any of the top $10$ distances tolerate noise to a certain degree. Moreover, the results show that some distances are less affected by the added noise comparing to other distances.

Motivation & Objective

• Assess how the choice of distance/similarity measure impacts KNN classification accuracy, precision, and recall across real-world datasets.
• Compare a large suite of distance measures (across eight families and 54 metrics) to identify top performers.
• Evaluate KNN robustness to noise by examining performance degradation with increasing noise levels.
• Identify distance metrics that are less sensitive to noise and thus more robust for KNN classification.

Proposed method

• Compile and categorize 54 distance/similarity measures into eight major families.
• Evaluate KNN performance (accuracy, precision, recall) on a large collection of real-world datasets with varying noise levels.
• Analyze robustness by adding different noise levels up to 90% and observing performance changes.
• Identify the top-performing distances and compare their noise tolerance.
• Summarize how results compare across datasets and discuss implications for practice.

Experimental results

Research questions

• RQ1 What is the best distance metric to maximize KNN accuracy (and related metrics) across diverse datasets?
• RQ2 What distance metric maintains the best performance when noise is present in the data?
• RQ3 Which distance measures show the most robustness to noise in KNN classification?

Key findings

• Performance of KNN significantly depends on the distance metric used, with large gaps between distances.
• A recently proposed non-convex distance often yields the best results on most datasets.
• The top-performing distance degrades by about 20% when noise reaches 90%, indicating substantial robustness.
• Many of the top 10 distances tolerate noise to some degree.
• Some distances exhibit greater resistance to added noise than others, highlighting varying robustness across metrics.
• Hassanat distance is highlighted as a strong performer in multiple comparisons.