[Paper Review] Subspace Determination through Local Intrinsic Dimensional Decomposition: Theory and Experimentation

Ruben Becker, Imane Hafnaoui+3 more|arXiv (Cornell University)|Jul 15, 2019

Image and Object Detection Techniques45 references4 citations

TL;DR

This paper proposes a novel method for identifying locally relevant features in high-dimensional data using Local Intrinsic Dimensionality (LID) decomposition along axis projections. By estimating LID per feature and leveraging its additive properties, the approach detects axis-aligned subspaces that support cluster formation, outperforming baseline methods like DiSH in subspace detection accuracy on synthetic manifolds with higher intrinsic dimensions.

ABSTRACT

Axis-aligned subspace clustering generally entails searching through enormous numbers of subspaces (feature combinations) and evaluation of cluster quality within each subspace. In this paper, we tackle the problem of identifying subsets of features with the most significant contribution to the formation of the local neighborhood surrounding a given data point. For each point, the recently-proposed Local Intrinsic Dimension (LID) model is used in identifying the axis directions along which features have the greatest local discriminability, or equivalently, the fewest number of components of LID that capture the local complexity of the data. In this paper, we develop an estimator of LID along axis projections, and provide preliminary evidence that this LID decomposition can indicate axis-aligned data subspaces that support the formation of clusters.

Motivation & Objective

• To address the challenge of identifying locally relevant features in high-dimensional data for subspace clustering.
• To develop a method that estimates feature-wise contribution to local data complexity using Local Intrinsic Dimensionality (LID).
• To provide a preliminary, independent step for subspace identification prior to clustering, enabling cluster-wise or point-wise feature relevance assessment.
• To evaluate the effectiveness of LID decomposition in detecting true subspaces compared to existing methods like DiSH.
• To explore the potential of LID decomposition as a new primitive for subspace clustering and related tasks.

Proposed method

• The paper introduces an estimator for LID decomposed along individual axis projections, compensating for bias introduced during projection.
• It leverages the theoretical property that LID estimates are additive across features under ideal conditions, using this to identify dominant features in local neighborhoods.
• A two-stage clustering pipeline is proposed: first, LID decomposition assigns points to relevant subspaces; second, DBSCAN is applied within these subspaces.
• The method uses a modified version of the Relative Non-Intersecting Area (RNIA) and introduces a new metric, Average Relative Relevance (ARR), to evaluate subspace detection performance.
• The approach is evaluated on synthetic datasets with known manifolds, including uniform, Gaussian, and curved manifolds, across varying intrinsic dimensions.
• Theoretical analysis and empirical validation are used to assess the additivity of LID estimators and their convergence properties in real-world settings.

Experimental results

Research questions

• RQ1 Can LID decomposition along axis projections effectively identify the most relevant features for local data structure in high-dimensional spaces?
• RQ2 How does the performance of LID-based subspace detection compare to existing methods like DiSH in terms of identifying true subspaces?
• RQ3 To what extent is the additivity of LID estimators preserved in practice, especially across different data distributions?
• RQ4 Can LID decomposition serve as a viable primitive for subspace clustering, particularly when cluster-specific feature relevance is required?
• RQ5 What metrics are most appropriate for evaluating subspace detection when ground-truth subspaces are known?

Key findings

• The LID decomposition method significantly outperformed DiSH on all four synthetic datasets, particularly on D4, which had the highest average manifold dimension.
• For D4, the LID-based approach achieved an RNIA score of 0.85 and an ARR of 0.78, indicating strong detection of true and relevant features.
• The method demonstrated robustness in detecting subspaces across diverse manifold types, including hypercubes, Gaussian distributions, and twisted manifolds like the Möbius band.
• The experimental results showed that MLE estimates of decomposed LID summed closely to the overall LID estimate, suggesting reasonable additivity under uniform distributions.
• The proposed Average Relative Relevance (ARR) metric effectively captured the efficiency of detecting the most relevant features, even when not all true features were recovered.
• The study provides preliminary evidence that LID decomposition can guide subspace clustering by identifying informative feature subsets prior to clustering execution.