[Paper Review] Mode-Seeking Clustering and Density Ridge Estimation via Direct Estimation of Density-Derivative-Ratios

Hiroaki Sasaki, Takafumi Kanamori+3 more|arXiv (Cornell University)|Jul 6, 2017

Yersinia bacterium, plague, ectoparasites research55 references3 citations

TL;DR

This paper proposes a direct estimator for density-derivative-ratio ratios, bypassing traditional three-step density estimation, and applies it to mode-seeking clustering and density ridge estimation. The method achieves improved convergence rates and outperforms existing approaches, especially in high-dimensional settings, by avoiding error propagation from intermediate density derivative estimation.

ABSTRACT

Modes and ridges of the probability density function behind observed data are useful geometric features. Mode-seeking clustering assigns cluster labels by associating data samples with the nearest modes, and estimation of density ridges enables us to find lower-dimensional structures hidden in data. A key technical challenge both in mode-seeking clustering and density ridge estimation is accurate estimation of the ratios of the first- and second-order density derivatives to the density. A naive approach takes a three-step approach of first estimating the data density, then computing its derivatives, and finally taking their ratios. However, this three-step approach can be unreliable because a good density estimator does not necessarily mean a good density derivative estimator, and division by the estimated density could significantly magnify the estimation error. To cope with these problems, we propose a novel estimator for the \emph{density-derivative-ratios}. The proposed estimator does not involve density estimation, but rather \emph{directly} approximates the ratios of density derivatives of any order. Moreover, we establish a convergence rate of the proposed estimator. Based on the proposed estimator, novel methods both for mode-seeking clustering and density ridge estimation are developed, and the respective convergence rates to the mode and ridge of the underlying density are also established. Finally, we experimentally demonstrate that the developed methods significantly outperform existing methods, particularly for relatively high-dimensional data.

Motivation & Objective

• To address the instability in traditional three-step methods that estimate density, then its derivatives, then their ratios, which amplifies estimation errors.
• To develop a direct estimator for ratios of density derivatives to the density, avoiding explicit density estimation.
• To establish theoretical convergence rates for the proposed estimator and its applications in clustering and ridge estimation.
• To improve performance in mode-seeking clustering and density ridge estimation, particularly in high-dimensional data.
• To demonstrate empirical superiority of the proposed methods over existing state-of-the-art techniques.

Proposed method

• The method directly estimates ratios of density derivatives of any order to the density using a kernel-based approach in a reproducing kernel Hilbert space (RKHS).
• It avoids the three-step process by constructing estimators that bypass density estimation, reducing error propagation.
• The estimator leverages the reproducing property of RKHS to approximate derivative ratios via inner products with kernel derivatives.
• Convergence rates are established for the proposed estimator under regularity conditions on the density and kernel.
• The method is applied to mode-seeking clustering via gradient ascent toward estimated modes and to density ridge estimation using projected gradient ascent in subspace constraints.
• A low-rank approximation with kernel centers is introduced to reduce computational cost while preserving accuracy.

Experimental results

Research questions

• RQ1 Can direct estimation of density-derivative-ratios improve the reliability of mode-seeking clustering and density ridge estimation compared to indirect three-step methods?
• RQ2 What is the theoretical convergence rate of the proposed direct estimator for density-derivative-ratios?
• RQ3 How does the proposed method perform in high-dimensional data where traditional methods often fail?
• RQ4 Can the direct estimator reduce computational cost without sacrificing clustering or ridge estimation accuracy?
• RQ5 Does the method achieve better performance than existing state-of-the-art approaches in real-world and synthetic datasets?

Key findings

• The proposed direct estimator for density-derivative-ratios achieves a convergence rate of $ O_P(n^{- ext{min}ig"){1/4, rac{ u}{2( u+1)}ig")}) $, where $ \nu $ is a smoothness parameter.
• The method significantly outperforms existing approaches in mode-seeking clustering, especially in high-dimensional data, as shown in experiments with three-Gaussian-blob datasets.
• For density ridge estimation, the proposed method achieves accurate recovery of lower-dimensional structures even in noisy, high-dimensional settings.
• The low-rank approximation (LSLDGC) reduces computation costs substantially while maintaining clustering performance with only a small number of kernel centers.
• Empirical results demonstrate that the method is robust to noise and maintains high accuracy across various data distributions and dimensions.
• Theoretical analysis confirms that the proposed estimator avoids error amplification from division by estimated density, a key limitation of traditional methods.