[Paper Review] Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)

Peter Sorrenson, Carsten Rother+1 more|arXiv (Cornell University)|Jan 14, 2020

Neural Networks and ApplicationsComputer Science26 references48 citations

TL;DR

The paper extends nonlinear ICA to unknown latent-dimension settings using a volume-preserving invertible network (GIN), and demonstrates recovery of informative latent variables, including 22 interpretable factors from EMNIST.

ABSTRACT

A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process. Recent breakthrough work by Khemakhem et al. (2019) on nonlinear ICA has answered this question for a broad class of conditional generative processes. We extend this important result in a direction relevant for application to real-world data. First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model. Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation. Second, we introduce a modification of the RealNVP invertible neural network architecture (Dinh et al. (2016)) which is particularly suitable for this type of problem: the General Incompressible-flow Network (GIN). Experiments on artificial data and EMNIST demonstrate that theoretical predictions are indeed verified in practice. In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.

Motivation & Objective

• Generalize nonlinear ICA identifiability to unknown intrinsic problem dimension.
• Introduce a volume-preserving invertible network (GIN) for nonlinear ICA tasks.
• Show that informative latent variables can be recovered and separated from noise.
• Demonstrate the theory and method on artificial data and the EMNIST dataset.

Proposed method

• Formulate nonlinear ICA with latent z and condition u, and a noisy space with x = f(z, ε).
• Use an estimating model g (an invertible neural network) with w = g^{-1}(x; θ) to map data back to latent space.
• Impose volume preservation by modifying RealNVP to ensure a unit Jacobian determinant in GIN.
• Assume conditional exponential-family distributions for z|u and w|u, with sufficient statistics T and T', and derive their affine relationship under identifiability (T(z) = A T'(w) + c).
• When latent spaces are Gaussian, show z_i corresponds to a single w_j up to translation and scaling, enabling intrinsic-dimension discovery.

Experimental results

Research questions

• RQ1 Can the intrinsic dimension of the generating latent space be unknown and still be recovered?
• RQ2 Under what conditions can informative latent variables be disentangled from noise in nonlinear ICA with conditioning?
• RQ3 Does a volume-preserving invertible network (GIN) enable recovering true latent variables up to simple transformations?
• RQ4 How does the theory extend to Gaussian (and two-parameter exponential-family) latent spaces in practice?

Key findings

• The theory extends nonlinear ICA to unknown intrinsic dimension, enabling discovery of the generating latent space dimension.
• A General Incompressible-flow Network (GIN) is proposed as a volume-preserving RealNVP variant for nonlinear ICA.
• In Gaussian latent spaces, each generating latent variable is recovered from exactly one estimated latent variable up to translation and scaling, with unused estimates encoding noise.
• Experiments on artificial data show that informative variables are separated from noise when mixture components overlap sufficiently.
• On EMNIST, GIN extracts 22 informative latent variables; eight global and fourteen local, with interpretable effects on digits.
• The learned latent variables stay interpretable despite deviations from theory assumptions, indicating practical viability of the approach.