[Paper Review] Non-Independent Components Analysis

Geert Mesters, Piotr Zwiernik|arXiv (Cornell University)|Jun 28, 2022

Blind Source Separation Techniques60 references4 citations

TL;DR

This paper proposes a general framework to identify the mixing matrix $ A $ in linear models $ AY = \varepsilon $ without assuming independent components, instead relying on zero restrictions on higher-order moment or cumulant tensors of $ \varepsilon $. It establishes that diagonal and reflectionally invariant tensor restrictions—such as setting off-diagonal or odd-index entries to zero—ensure identification of $ A $ up to sign and permutation, even when components are dependent, and introduces efficient minimum distance estimators that outperform traditional ICA methods under non-i.i.d. errors.

ABSTRACT

A seminal result in the ICA literature states that for $AY = \varepsilon$, if the components of $\varepsilon$ are independent and at most one is Gaussian, then $A$ is identified up to sign and permutation of its rows (Comon, 1994). In this paper we study to which extent the independence assumption can be relaxed by replacing it with restrictions on higher order moment or cumulant tensors of $\varepsilon$. We document new conditions that establish identification for several non-independent component models, e.g. common variance models, and propose efficient estimation methods based on the identification results. We show that in situations where independence cannot be assumed the efficiency gains can be significant relative to methods that rely on independence.

Motivation & Objective

• To relax the strong independence assumption in Independent Components Analysis (ICA) while preserving identifiability of the mixing matrix $ A $.
• To identify conditions on higher-order moments or cumulants of $ \varepsilon $ that ensure $ A $ is identifiable up to sign and permutation, even when components are dependent.
• To develop efficient estimation methods based on these relaxed conditions, particularly using generalized moment estimators with moment and cumulant restrictions.
• To demonstrate that these methods outperform standard ICA techniques in settings where independence does not hold, such as common variance or scale-elliptical models.
• To extend the applicability of ICA to real-world models where components are dependent but structured higher-order moments still allow identification.

Proposed method

• Uses zero restrictions on $ r $-th order moment or cumulant tensors of $ \varepsilon $, specifically diagonal and reflectionally invariant patterns, to identify $ A $.
• Proposes a class of higher-order minimum distance estimators that incorporate both moment and cumulant restrictions, generalizing existing methods like JADE.
• Employs generalized method of moments (GMM) principles with efficient weighting matrices $ \widehat{\Sigma}_n^{-1} $ to improve estimation efficiency.
• Applies the Darmois-Skitovich theorem and tensor algebra to prove identification under non-i.i.d. error structures, such as common variance or scale-elliptical distributions.
• Uses simulation studies with multiple scaled elliptical models to compare performance against standard ICA methods under dependent components.
• Establishes asymptotic normality and consistency of the proposed estimators under standard regularity conditions.

Experimental results

Research questions

• RQ1 Can the independence assumption in ICA be relaxed while still ensuring identifiability of the mixing matrix $ A $?
• RQ2 What specific zero patterns in higher-order moment or cumulant tensors of $ \varepsilon $ are sufficient to identify $ A $ up to sign and permutation?
• RQ3 How do identification and estimation performance compare between methods assuming independence and those assuming structured non-independence?
• RQ4 Can efficient estimation be achieved using moment and cumulant restrictions without assuming component independence?
• RQ5 Do reflectionally invariant tensor restrictions—where only entries with even index repetition are non-zero—still allow identification when odd-order tensors vanish due to symmetry?

Key findings

• Diagonal tensor restrictions (off-diagonal entries zero) in any $ r $-th order moment or cumulant tensor identify $ A $ up to sign and permutation, even without independence.
• Reflectionally invariant restrictions—non-zero entries only when each index appears an even number of times—also ensure identification and strictly relax the diagonal assumption.
• The proposed minimum distance estimators based on these restrictions are consistent and asymptotically normal under standard regularity conditions.
• In simulations with dependent components (e.g., multiple scaled elliptical errors), the new methods outperform standard ICA techniques like FastICA, JADE, and TICA, with Amari errors reduced by up to 30%.
• Using the efficient weighting matrix $ \widehat{\Sigma}_n^{-1} $ improves performance but is sensitive to estimation accuracy; in practice, $ I_d $ often performs comparably or better.
• The framework applies to models like common variance, scale-elliptical, and mean-independent components, for which no prior identification results existed under non-i.i.d. errors.