[Paper Review] Fast and Extensible Online Multivariate Kernel Density Estimation

Jaime Puyana Ferreira, David Martins de Matos+1 more|arXiv (Cornell University)|Jun 8, 2016

Gaussian Processes and Bayesian Inference3 citations

TL;DR

xokde++ is a fast, numerically robust, and extensible online multivariate kernel density estimation method that uses Gaussian mixture models with diagonal covariance matrices to achieve up to 40x speedup and 90% memory reduction over state-of-the-art approaches, while maintaining comparable or better accuracy, especially on high-dimensional and non-normalized data.

ABSTRACT

We present xokde++, a state-of-the-art online kernel density estimation approach that maintains Gaussian mixture models input data streams. The approach follows state-of-the-art work on online density estimation, but was redesigned with computational efficiency, numerical robustness, and extensibility in mind. Our approach produces comparable or better results than the current state-of-the-art, while achieving significant computational performance gains and improved numerical stability. The use of diagonal covariance Gaussian kernels, which further improve performance and stability, at a small loss of modelling quality, is also explored. Our approach is up to 40 times faster, while requiring 90\% less memory than the closest state-of-the-art counterpart.

Motivation & Objective

• Address the computational and numerical instability challenges in online multivariate kernel density estimation (KDE) for evolving data streams.
• Improve upon existing online KDE methods like oKDE by enhancing computational efficiency, memory usage, and numerical robustness.
• Enable high-dimensional data processing by reducing the complexity of covariance estimation through diagonal approximations.
• Design a modular, extensible C++ implementation that supports easy experimentation with different covariance structures and model components.
• Achieve state-of-the-art performance in accuracy, speed, and memory efficiency for online density estimation tasks.

Proposed method

• Adapts the oKDE framework by maintaining a two-level model: a compressed Gaussian mixture model (GMM) for efficiency and a detailed observation model for component recovery.
• Uses diagonal covariance matrices in Gaussian components to reduce computational complexity and improve numerical stability, especially in high dimensions.
• Employs a hierarchical compression strategy with component merging and splitting to maintain model complexity and accuracy without storing all past samples.
• Applies optimal bandwidth selection via a scalable approximation of the mean integrated squared error (MISE) using component-wise statistics and matrix operations.
• Implements numerical stability improvements such as avoiding singular covariance matrices and using pseudo-inverses and pseudo-log-determinants for ill-conditioned cases.
• Leverages C++ templates and modern libraries to enable extensibility, allowing plug-and-play alternatives like triangular or low-rank covariance approximations.

Experimental results

Research questions

• RQ1 Can online KDE be made significantly faster and more memory-efficient while preserving or improving model accuracy?
• RQ2 How does using diagonal covariance matrices affect model quality and computational performance in high-dimensional online KDE?
• RQ3 To what extent can numerical instability in online KDE be mitigated through improved matrix handling and covariance regularization?
• RQ4 Can a modular, extensible C++ implementation support diverse modeling choices (e.g., different covariance types) without sacrificing performance?
• RQ5 How does xokde++ compare to oKDE and other state-of-the-art methods in terms of accuracy, speed, and memory usage across diverse real-world datasets?

Key findings

• xokde++ achieves up to 40x speedup and 90% memory reduction compared to the closest state-of-the-art method, with comparable or better accuracy.
• The use of diagonal covariance matrices reduces average memory usage to 73–78.5% of full covariance models, with only a 0.72–1.67% absolute accuracy loss on average.
• xokde++ improves classifier accuracy by 6.5% on average (to 78%) compared to oKDE (73%), particularly excelling on non-normalized and high-dimensional data.
• The method maintains numerical stability even with ill-conditioned or degenerate covariance matrices, avoiding failures common in other approaches.
• The implementation is highly extensible: switching from full to diagonal covariances required minimal code changes and delivered significant performance gains.
• On the Skin dataset, xokde++ achieved 99.6% accuracy with only 8 components, while using 83.1 MB memory, and reduced time from 572.9s to 192.2s with diagonal covariances.