[Paper Review] A Bayesian reassessment of nearest-neighbour classification

Lionel Cucala, Jean‐Michel Marin+2 more|ArXiv.org|Feb 10, 2008

Bayesian Methods and Mixture Models16 references4 citations

TL;DR

This paper proposes a fully Bayesian re-interpretation of the k-nearest-neighbour (k-NN) classification method by constructing a coherent probabilistic model that embeds k-NN within a proper statistical framework. It introduces a novel MCMC approach using perfect sampling or Gibbs sampling to handle the intractable normalising constant, demonstrating that pseudo-likelihood approximations are unreliable for moderate-sized datasets, while the proposed Bayesian approach improves classification accuracy and uncertainty quantification over standard k-NN.

ABSTRACT

The k-nearest-neighbour procedure is a well-known deterministic method used in supervised classification. This paper proposes a reassessment of this approach as a statistical technique derived from a proper probabilistic model; in particular, we modify the assessment made in a previous analysis of this method undertaken by Holmes and Adams (2002,2003), and evaluated by Manocha and Girolami (2007), where the underlying probabilistic model is not completely well-defined. Once a clear probabilistic basis for the k-nearest-neighbour procedure is established, we derive computational tools for conducting Bayesian inference on the parameters of the corresponding model. In particular, we assess the difficulties inherent to pseudo-likelihood and to path sampling approximations of an intractable normalising constant, and propose a perfect sampling strategy to implement a correct MCMC sampler associated with our model. If perfect sampling is not available, we suggest using a Gibbs sampling approximation. Illustrations of the performance of the corresponding Bayesian classifier are provided for several benchmark datasets, demonstrating in particular the limitations of the pseudo-likelihood approximation in this set-up.

Motivation & Objective

• To address the lack of a well-defined probabilistic foundation for the k-NN method, which is traditionally deterministic and lacks error assessment.
• To develop a coherent Bayesian model that treats k-NN as a statistical inference problem rather than a heuristic algorithm.
• To provide a principled framework for estimating classification uncertainty and model parameters, including the number of neighbours k.
• To evaluate and compare computational methods for handling the intractable normalising constant in the model, particularly pseudo-likelihood, path sampling, and perfect sampling.
• To demonstrate that the proposed Bayesian k-NN approach outperforms standard k-NN in terms of classification error and uncertainty quantification on benchmark datasets.

Proposed method

• Formulates a joint probabilistic model over training data that embeds the k-NN procedure as a Boltzmann-type model with compatible conditionals.
• Derives a proper likelihood function based on the k-NN neighbourhood structure, enabling full Bayesian inference.
• Addresses the intractable normalising constant using three computational strategies: pseudo-likelihood, path sampling, and perfect sampling via the Møller et al. algorithm.
• Proposes a practical Gibbs sampling approximation as a viable alternative to perfect sampling when computational cost is prohibitive.
• Employs MCMC sampling (perfect or Gibbs) to estimate posterior distributions over class labels and model parameters, including k.
• Uses predictive maps to visualize classification uncertainty and decision boundaries, enhancing interpretability.

Experimental results

Research questions

• RQ1 Can the k-NN classification method be formally embedded within a coherent Bayesian probabilistic model?
• RQ2 How do different approximations for the intractable normalising constant—pseudo-likelihood, path sampling, and perfect sampling—affect inference quality in k-NN?
• RQ3 Does the proposed Bayesian k-NN approach improve classification accuracy and uncertainty quantification compared to standard k-NN?
• RQ4 Is the Gibbs sampling approximation a practical and reliable alternative to perfect sampling in this context?
• RQ5 To what extent does the Bayesian framework allow for joint inference on k and class membership with uncertainty estimates?

Key findings

• The pseudo-likelihood approximation is found to be unreliable for moderate-sized datasets, leading to poor inference in the k-NN context.
• Perfect sampling and path sampling provide valid but computationally expensive alternatives, with path sampling being useful for validation but not practical for routine use.
• The proposed Gibbs sampling approximation is both computationally feasible and effective, offering a practical alternative to perfect sampling.
• On the Pima Indian diabetes dataset, the Bayesian k-NN achieved a prediction error of 0.209, comparable to the best k-NN results (e.g., k=57–66 with error 0.205–0.208).
• On the forensic glass fragments dataset, the Bayesian approach reduced error from 0.35 (standard k-NN with k=17) to 0.29, demonstrating a substantial improvement.
• The Bayesian framework enables visualization of classification uncertainty through predictive maps, revealing regions of ambiguity between classes.