[Paper Review] KNN Ensembles for Tweedie Regression: The Power of Multiscale Neighborhoods

Colleen Farrelly|arXiv (Cornell University)|Jul 29, 2017

Topological and Geometric Data Analysis40 references5 citations

TL;DR

This paper proposes KNN ensemble methods that leverage multiscale neighborhoods by varying k and combining bagged features and observations for Tweedie regression. It demonstrates that varying k significantly improves prediction beyond bagging features or samples, with strong performance gains over standard KNN and state-of-the-art models on both simulated and real datasets, particularly in high-dimensional settings.

ABSTRACT

Very few K-nearest-neighbor (KNN) ensembles exist, despite the efficacy of this approach in regression, classification, and outlier detection. Those that do exist focus on bagging features, rather than varying k or bagging observations; it is unknown whether varying k or bagging observations can improve prediction. Given recent studies from topological data analysis, varying k may function like multiscale topological methods, providing stability and better prediction, as well as increased ensemble diversity. This paper explores 7 KNN ensemble algorithms combining bagged features, bagged observations, and varied k to understand how each of these contribute to model fit. Specifically, these algorithms are tested on Tweedie regression problems through simulations and 6 real datasets; results are compared to state-of-the-art machine learning models including extreme learning machines, random forest, boosted regression, and Morse-Smale regression. Results on simulations suggest gains from varying k above and beyond bagging features or samples, as well as the robustness of KNN ensembles to the curse of dimensionality. KNN regression ensembles perform favorably against state-of-the-art algorithms and dramatically improve performance over KNN regression. Further, real dataset results suggest varying k is a good strategy in general (particularly for difficult Tweedie regression problems) and that KNN regression ensembles often outperform state-of-the-art methods. These results for k-varying ensembles echo recent theoretical results in topological data analysis, where multidimensional filter functions and multiscale coverings provide stability and performance gains over single-dimensional filters and single-scale covering. This opens up the possibility of leveraging multiscale neighborhoods and multiple measures of local geometry in ensemble methods.

Motivation & Objective

• To investigate the impact of varying k, bagging features, and bagging observations in KNN ensembles for Tweedie regression.
• To evaluate whether multiscale neighborhood approaches—inspired by topological data analysis—improve model stability and prediction accuracy.
• To compare the performance of KNN ensemble variants against state-of-the-art machine learning models on real and simulated Tweedie regression problems.
• To assess the robustness of KNN ensembles to the curse of dimensionality in high-dimensional settings.

Proposed method

• Seven KNN ensemble algorithms are developed, combining bagged features, bagged observations, and varied k values across multiple runs.
• Each ensemble uses a voting or averaging strategy over predictions from multiple KNN models with different k values or bootstrap samples.
• The method incorporates multiscale neighborhoods by systematically varying k across a range of values, mimicking multiscale topological analysis.
• Ensemble diversity is enhanced through combinations of feature bagging, observation bagging, and k-varying strategies.
• Models are trained and evaluated on Tweedie regression tasks using metrics like mean squared error and deviance.
• The approach is validated through simulations and six real-world datasets, with comparisons to extreme learning machines, random forests, boosted regression, and Morse-Smale regression.

Experimental results

Research questions

• RQ1 Does varying k in KNN ensembles lead to better prediction performance than fixed k or standard bagging techniques?
• RQ2 How do combinations of bagged features, bagged observations, and varied k contribute to model fit and robustness in Tweedie regression?
• RQ3 Can multiscale neighborhood strategies in KNN ensembles improve performance and stability, particularly in high-dimensional or complex data settings?
• RQ4 How do KNN ensemble models compare to state-of-the-art algorithms like random forests and extreme learning machines on Tweedie regression tasks?
• RQ5 Is the performance gain from k-varying ensembles robust across diverse real-world datasets and simulation scenarios?

Key findings

• Varying k in KNN ensembles provides significant performance gains over both fixed-k KNN and ensembles that only bag features or observations.
• KNN ensemble models outperform standard KNN regression and achieve competitive or superior results compared to state-of-the-art models like random forests and extreme learning machines.
• The proposed ensembles demonstrate robustness to the curse of dimensionality, maintaining strong performance in high-dimensional data settings.
• Results from real datasets indicate that k-varying ensembles are particularly effective for difficult Tweedie regression problems, often surpassing state-of-the-art methods.
• The performance gains align with theoretical insights from topological data analysis, supporting the use of multiscale neighborhoods and multiple local geometry measures in ensemble learning.
• The study confirms that combining k-varying strategies with bagging enhances diversity and prediction accuracy, validating the multiscale neighborhood approach as a powerful framework.