[Paper Review] On the existence of Optimal Subspace Clustering Models

Akram Aldroubi, Romain Tessera|arXiv (Cornell University)|Aug 27, 2010

Bayesian Methods and Mixture Models19 references3 citations

TL;DR

This paper establishes necessary and sufficient conditions for the existence of optimal subspace clustering models in both finite- and infinite-dimensional Hilbert spaces. It introduces the concept of a contact hull in infinite dimensions and proves that optimal approximations exist when the family of subspaces is closed under certain topological conditions, generalizing the Eckart-Young theorem and providing a new proof for shift-invariant subspace clustering.

ABSTRACT

Given a set of vectors $\F=\{f_1,\dots,f_m\}$ in a Hilbert space $\HH$, and given a family $\CC$ of closed subspaces of $\HH$, the {\it subspace clustering problem} consists in finding a union of subspaces in $\CC$ that best approximates (models) the data $\F$. This problem has applications and connections to many areas of mathematics, computer science and engineering such as the Generalized Principle Component Analysis (GPCA), learning theory, compressed sensing, and sampling with finite rate of innovation. In this paper, we characterize families of subspaces $\CC$ for which such a best approximation exists. In finite dimensions the characterization is in terms of the convex hull of an augmented set $\CC^+$. In infinite dimensions however, the characterization is in terms of a new but related notion of contact hull. As an application, the existence of best approximations from $π(G)$-invariant families $\CC$ of unitary representations of abelian groups is derived.

Motivation & Objective

• To determine necessary and sufficient conditions for the existence of a best approximation to a finite data set by a union of subspaces from a given family.
• To extend the classical Eckart-Young theorem on low-rank approximation to more general families of subspaces, including infinite-dimensional settings.
• To provide a topological characterization of families of subspaces that guarantee the existence of optimal clustering models.
• To establish the Minimum Subspace Approximation Property (MSAP) for invariant families under unitary group representations, particularly shift-invariant spaces.
• To generalize prior results on subspace clustering in compressed sensing and hybrid linear models by introducing a new theoretical framework based on projectors and spectral decompositions.

Proposed method

• Reformulate the non-linear least squares subspace approximation problem as a minimization over families of closed subspaces.
• Introduce the augmented family $\mathcal{C}^+$ and define its convex hull in finite dimensions to characterize MSAP.
• In infinite dimensions, define the contact hull as a generalization of the convex hull to characterize MSAP.
• Use spectral decomposition of projectors in terms of direct integrals over the dual group $\hat{G}$ for unitary representations.
• Prove that $\pi(G)$-invariant subspaces with bounded corank admit measurable families of generators, ensuring existence of minimizers.
• Apply the theory to prove MSAP for shift-invariant subspaces in $L^2(\mathbb{R})$ and $\ell^2(\mathbb{Z})$, and for finite cyclic group actions.

Experimental results

Research questions

• RQ1 Under what conditions on a family $\mathcal{C}$ of closed subspaces does a best approximation exist for any finite data set $\mathbf{F}$?
• RQ2 How can the classical Eckart-Young theorem be generalized to infinite-dimensional Hilbert spaces and non-convex families of subspaces?
• RQ3 What topological or algebraic structure ensures the existence of optimal subspace clustering models in the presence of non-linear constraints?
• RQ4 Can the existence of optimal approximations be guaranteed for $\pi(G)$-invariant subspaces under unitary group representations?
• RQ5 What is the role of the dual group $\hat{G}$ and direct integral decompositions in characterizing optimal clustering models?

Key findings

• In finite-dimensional Hilbert spaces, a family $\mathcal{C}$ of closed subspaces satisfies MSAP if and only if the convex hull of its augmented set $\mathcal{C}^+$ contains the optimal projector.
• In infinite-dimensional Hilbert spaces, the existence of optimal approximations is characterized by the contact hull of $\mathcal{C}$, a new topological construction generalizing convex hulls.
• The family of all subspaces of dimension $\leq r$ in a Hilbert space satisfies MSAP, generalizing the Eckart-Young theorem to infinite dimensions.
• For $\pi(G)$-invariant subspaces of $L^2(\mathbb{R})$ with bounded corank, the existence of optimal approximations is guaranteed, providing a new proof of a result in [ACHM07].
• The set of all $\pi(G)$-invariant subspaces of dimension $\leq k$ in $\ell^2(\mathbb{Z})$ or $\ell^2(\{1,\dots,d\})$ satisfies MSAP when $G$ is a discrete abelian group.
• The existence of measurable, orthonormal bases for kernels of projectors in direct integral decompositions ensures the existence of optimal clustering models in invariant settings.