[Paper Review] Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

Aapo Hyvärinen, Hiroaki Sasaki+1 more|arXiv (Cornell University)|May 22, 2018

Blind Source Separation Techniques25 references41 citations

TL;DR

The paper introduces a general nonlinear ICA framework using an auxiliary variable to induce identifiability, and empirically validates a contrastive-learning-based estimator that recovers latent components.

ABSTRACT

Nonlinear ICA is a fundamental problem for unsupervised representation learning, emphasizing the capacity to recover the underlying latent variables generating the data (i.e., identifiability). Recently, the very first identifiability proofs for nonlinear ICA have been proposed, leveraging the temporal structure of the independent components. Here, we propose a general framework for nonlinear ICA, which, as a special case, can make use of temporal structure. It is based on augmenting the data by an auxiliary variable, such as the time index, the history of the time series, or any other available information. We propose to learn nonlinear ICA by discriminating between true augmented data, or data in which the auxiliary variable has been randomized. This enables the framework to be implemented algorithmically through logistic regression, possibly in a neural network. We provide a comprehensive proof of the identifiability of the model as well as the consistency of our estimation method. The approach not only provides a general theoretical framework combining and generalizing previously proposed nonlinear ICA models and algorithms, but also brings practical advantages.

Motivation & Objective

• Unify and extend identifiability conditions for nonlinear ICA by introducing an auxiliary variable that modulates component distributions.
• Provide a practical learning algorithm based on contrastive (logistic regression) discrimination between real and randomized augmented data.
• Prove consistency of the estimator and establish identifiability under broad conditions.
• Show how time, history, and class labels can serve as auxiliary variables to recover latent sources.

Proposed method

• Define a nonlinear ICA model where latent components s_i depend on an observed auxiliary variable u and are conditionally independent given u.
• Construct two datasets by augmenting data with (x,u) versus (x,u*) where u* is randomized, and train a nonlinear logistic regression to discriminate between them.
• Use a regression function r(x,u) with hidden representations h_i(x) and feature maps psi_i to estimate posteriors.
• Prove consistency of the estimator under regularity conditions and derive identifiability results for general and conditionally exponential source families.
• Show how different definitions of u (time, history, segments, class labels) fit into the framework and link to TCL, PCL, and related methods.

Experimental results

Research questions

• RQ1 Can an auxiliary variable u render nonlinear ICA identifiable for general nonlinear mixtures?
• RQ2 What are the exact identifiability and consistency conditions for the proposed augmented-contrastive learning approach?
• RQ3 How do different choices of auxiliary variables (time, history, nonstationary segments, class labels) affect identifiability and practical recovery of latent sources?
• RQ4 Does the logistic regression-based estimator recover the independent components up to invertible component-wise transformations under the theory?
• RQ5 How does the framework relate to and generalize prior nonlinear ICA approaches such as TCL and PCL?

Key findings

• A general nonlinear ICA model with an auxiliary variable achieves identifiability under mild smoothness and variability conditions.
• A practical, consistent estimator is obtained via contrastive learning that discriminates augmented real data from randomized augmentations.
• The framework subsumes and generalizes time-contrastive learning (TCL) and permutation-contrastive learning (PCL) as special cases.
• In the conditional exponential case, the theory yields a linear indeterminacy similar to TCL, while in the general case the components are recovered up to component-wise invertible nonlinearities.
• Using class labels as auxiliary variables can avoid linear indeterminacy and enables supervised/unsupervised fusion for recovering latent sources.
• Simulation studies show performance comparable to TCL for nonstationary and temporally dependent sources.