[Paper Review] PIDForest: Anomaly Detection via Partial Identification

Parikshit Gopalan, Vatsal Sharan+1 more|arXiv (Cornell University)|Dec 8, 2019

Anomaly Detection Techniques and Applications4 citations

TL;DR

PIDForest is a novel anomaly detection algorithm that uses a geometric measure called PIDScore to identify anomalies based on their sparsity in subcubes of the feature space. By optimizing for high-variance splits and using sparsity-based leaf labeling in a random forest framework, it outperforms six benchmark methods on 12 real-world datasets, excelling in robustness to noise and irrelevant features while providing interpretable explanations via critical feature ranges.

ABSTRACT

We consider the problem of detecting anomalies in a large dataset. We propose a framework called Partial Identification which captures the intuition that anomalies are easy to distinguish from the overwhelming majority of points by relatively few attribute values. Formalizing this intuition, we propose a geometric anomaly measure for a point that we call PIDScore, which measures the minimum density of data points over all subcubes containing the point. We present PIDForest: a random forest based algorithm that finds anomalies based on this definition. We show that it performs favorably in comparison to several popular anomaly detection methods, across a broad range of benchmarks. PIDForest also provides a succinct explanation for why a point is labelled anomalous, by providing a set of features and ranges for them which are relatively uncommon in the dataset.

Motivation & Objective

• To address the challenge of detecting anomalies in high-dimensional, heterogeneous, and unlabeled datasets common in large-scale systems like data centers.
• To provide a clear, geometric definition of anomaly distinct from algorithmic implementation, enabling interpretability and principled detection.
• To improve upon Isolation Forest by making it robust to irrelevant and noisy features through variance-based split selection.
• To develop a scalable, efficient, and interpretable anomaly detection method that does not rely on generative modeling or distance metrics.
• To demonstrate superior performance and stability across diverse real and synthetic datasets under varying hyperparameters.

Proposed method

• Proposes PIDScore as a geometric anomaly measure: the maximum sparsity (volume/point count) over all subcubes containing a given point.
• Designs PIDForest as a random forest where splits are chosen to maximize variance in feature distributions, favoring informative, non-uniform features.
• Labels each leaf node with the sparsity of the subcube it represents, and computes the anomaly score as the maximum sparsity across all leaves a point reaches.
• Uses a greedy, randomized tree-building process that prioritizes features with high marginal variance to enhance detection of relevant attributes.
• Employs a subcube-based density estimation strategy that avoids reliance on distance metrics, making it unit-agnostic and suitable for mixed-type data.
• Introduces a robustness mechanism where hyperparameters (e.g., number of buckets, trees, depth) have minimal impact on performance beyond a moderate threshold.

Experimental results

Research questions

• RQ1 Can a geometric, density-based anomaly measure like PIDScore provide a principled and interpretable definition of anomalies independent of algorithmic implementation?
• RQ2 How does PIDForest’s variance-based splitting strategy improve robustness to irrelevant and noisy features compared to random or range-based splitting in Isolation Forest?
• RQ3 To what extent does PIDForest outperform existing anomaly detection algorithms across diverse real-world and synthetic datasets?
• RQ4 How stable is PIDForest’s performance under variations in key hyperparameters such as number of trees, depth, and number of buckets?
• RQ5 Can PIDForest provide meaningful, human-interpretable explanations by identifying specific feature ranges that contribute to anomaly scores?

Key findings

• PIDForest achieved the best performance in 6 out of 12 real-world benchmark datasets, outperforming six popular anomaly detection algorithms, with no other method ranking first in more than three datasets.
• The algorithm demonstrated strong resilience to noise and irrelevant attributes, maintaining high performance even when up to 50% of features were irrelevant.
• Hyperparameter sensitivity analysis showed that performance stabilized at moderate values of k (number of buckets), m (number of samples), t (number of trees), and h (tree depth), with minimal variation beyond these thresholds.
• PIDForest provided interpretable explanations by identifying specific feature ranges that contributed to high anomaly scores, enhancing troubleshooting utility in real-world monitoring systems.
• The algorithm significantly outperformed Isolation Forest on datasets with high-dimensional, heterogeneous features, particularly when irrelevant or noisy dimensions were present.
• In synthetic time-series experiments, PIDForest maintained high precision and recall across varying noise levels and data distributions, confirming its robustness and generalization capability.