[Paper Review] GBHT: Gradient Boosting Histogram Transform for Density Estimation

Jingyi Cui, Hanyuan Hang+2 more|arXiv (Cornell University)|Jun 10, 2021

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes GBHT, a novel gradient boosting algorithm for density estimation that uses negative log-likelihood as the loss function to enable boosting in unsupervised learning. By combining random histogram transforms with iterative gradient boosting, GBHT achieves faster convergence rates than its base learners, with theoretical guarantees in Hölder spaces $C^{0,eta}$ and $C^{1,eta}$, and demonstrates strong performance in anomaly detection on real-world datasets.

ABSTRACT

In this paper, we propose a density estimation algorithm called extit{Gradient Boosting Histogram Transform} (GBHT), where we adopt the extit{Negative Log Likelihood} as the loss function to make the boosting procedure available for the unsupervised tasks. From a learning theory viewpoint, we first prove fast convergence rates for GBHT with the smoothness assumption that the underlying density function lies in the space $C^{0,α}$. Then when the target density function lies in spaces $C^{1,α}$, we present an upper bound for GBHT which is smaller than the lower bound of its corresponding base learner, in the sense of convergence rates. To the best of our knowledge, we make the first attempt to theoretically explain why boosting can enhance the performance of its base learners for density estimation problems. In experiments, we not only conduct performance comparisons with the widely used KDE, but also apply GBHT to anomaly detection to showcase a further application of GBHT.

Motivation & Objective

• To develop a boosting-based method for unsupervised density estimation, overcoming the lack of theoretical foundations in existing boosting approaches for this task.
• To establish a theoretical framework explaining why boosting improves performance in density estimation, particularly by analyzing convergence rates in Hölder spaces.
• To design a practical algorithm that combines random histogram transforms with gradient boosting to enhance density estimation accuracy and smoothness.
• To validate GBHT's effectiveness through empirical comparisons with KDE and application in anomaly detection on real datasets.

Proposed method

• GBHT uses negative log-likelihood as the loss function, enabling gradient boosting in unsupervised density estimation.
• At each iteration, random histogram transforms are applied—comprising random rotations, scalings, and translations—to partition the input space into non-overlapping cells.
• Piecewise constant density estimators are learned on each transformed cell, forming the base learners.
• The algorithm iteratively improves the density estimate by minimizing empirical negative log-likelihood using a gradient descent adaptation of boosting.
• After training, the final density estimate is obtained by inverse-transforming the partitioned space back to the original input space.
• Nesterov’s acceleration is incorporated to improve convergence speed during the boosting process.

Experimental results

Research questions

• RQ1 Can boosting be theoretically justified for density estimation using an unsupervised loss function?
• RQ2 What are the convergence rates of GBHT under smoothness assumptions in Hölder spaces $C^{0,eta}$ and $C^{1,eta}$?
• RQ3 Does GBHT achieve better convergence rates than its base learner in the $C^{1,eta}$ space?
• RQ4 How does GBHT compare to traditional methods like KDE in real-world density estimation and anomaly detection tasks?
• RQ5 Can GBHT be effectively applied to downstream tasks such as anomaly detection with competitive performance?

Key findings

• GBHT achieves a fast convergence rate of $O(n^{-2(1+eta)/(4(1+eta)+d)})$ in the $C^{1,eta}$ space, which is strictly better than the lower bound $O(n^{-2/(2+d)})$ of the base histogram transform estimator.
• In the $C^{0,eta}$ space, GBHT achieves a fast convergence rate of $O(n^{-2eta/(2eta+d)})$, establishing theoretical consistency for the method.
• Empirical results show GBHT outperforms KDE and other baselines on 20 real-world datasets in anomaly detection, with AUC scores consistently among the top two across all datasets.
• GBHT achieves the best or second-best AUC in 18 out of 20 datasets, with a rank sum of 57, indicating superior overall performance compared to iForest, k-NN, LOF, and OCSVM.
• The method demonstrates robustness to varying data dimensionality and sample size, with strong performance on high-dimensional and imbalanced datasets.
• The integration of Nesterov’s acceleration improves training efficiency without compromising estimation accuracy.