[Paper Review] Canonical normalizing flows for manifold learning

Kyriakos Flouris, Ender Konukoğlu|arXiv (Cornell University)|Oct 19, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Canonical Manifold Flow (CMF), a novel optimization objective for manifold learning normalizing flows that enforces a sparse, non-degenerate, and locally orthogonal intrinsic basis in the latent space by minimizing the ℓ₁-norm of off-diagonal metric tensor elements. The method improves sample quality and latent space efficiency without additional computational cost, achieving lower FID scores than prior methods.

ABSTRACT

Manifold learning flows are a class of generative modelling techniques that assume a low-dimensional manifold description of the data. The embedding of such a manifold into the high-dimensional space of the data is achieved via learnable invertible transformations. Therefore, once the manifold is properly aligned via a reconstruction loss, the probability density is tractable on the manifold and maximum likelihood can be used to optimize the network parameters. Naturally, the lower-dimensional representation of the data requires an injective-mapping. Recent approaches were able to enforce that the density aligns with the modelled manifold, while efficiently calculating the density volume-change term when embedding to the higher-dimensional space. However, unless the injective-mapping is analytically predefined, the learned manifold is not necessarily an efficient representation of the data. Namely, the latent dimensions of such models frequently learn an entangled intrinsic basis, with degenerate information being stored in each dimension. Alternatively, if a locally orthogonal and/or sparse basis is to be learned, here coined canonical intrinsic basis, it can serve in learning a more compact latent space representation. Toward this end, we propose a canonical manifold learning flow method, where a novel optimization objective enforces the transformation matrix to have few prominent and non-degenerate basis functions. We demonstrate that by minimizing the off-diagonal manifold metric elements $\ell_1$-norm, we can achieve such a basis, which is simultaneously sparse and/or orthogonal. Canonical manifold flow yields a more efficient use of the latent space, automatically generating fewer prominent and distinct dimensions to represent data, and a better approximation of target distributions than other manifold flow methods in most experiments we conducted, resulting in lower FID scores.

Motivation & Objective

• To address the inefficiency and degeneracy in latent space representations of existing manifold learning flows (MLFs), where information is redundantly stored across dimensions.
• To improve the compactness and interpretability of the learned latent space by encouraging a canonical intrinsic basis—simultaneously sparse and locally orthogonal.
• To achieve this through a new optimization objective that leverages existing Jacobian computations, avoiding additional computational overhead.
• To demonstrate that a more structured latent basis leads to better generative performance and improved approximation of target data distributions.
• To provide a theoretically grounded, computationally efficient method that enhances manifold flow modeling while preserving exact likelihood computation.

Proposed method

• The method introduces a new loss term that minimizes the ℓ₁-norm of the off-diagonal elements of the Riemannian metric tensor derived from the flow’s Jacobian.
• This loss encourages the learned tangent vectors in the latent space to be sparse and approximately orthogonal, promoting a canonical basis without enforcing strict orthogonality.
• The optimization is integrated into the standard likelihood maximization objective of manifold learning flows, reusing Jacobian computations from the change-of-variable formula.
• The approach is compatible with existing MLF frameworks such as RNF and MFlow, and can be applied to both synthetic and image data.
• The method does not require additional invertibility constraints or architectural changes, making it lightweight and plug-and-play.
• The canonical basis emerges naturally from the optimization, enabling better disentanglement and reduced degeneracy in latent dimensions.

(a) Density plot for a fuzzy line learned with RNF, via Equation 4 .

Experimental results

Research questions

• RQ1 Can minimizing off-diagonal metric elements in the latent space lead to a more compact and non-degenerate representation in manifold learning flows?
• RQ2 Does enforcing sparsity and local orthogonality in the learned basis improve sample quality and likelihood approximation?
• RQ3 Can this be achieved without introducing additional computational cost or architectural complexity?
• RQ4 How does the resulting canonical basis compare to standard MLF methods in terms of FID scores and latent space disentanglement?
• RQ5 To what extent does the method generalize across different data types, including synthetic and image data?

Key findings

• The proposed CMF method achieves lower Fréchet Inception Distance (FID) scores than baseline manifold learning flows across multiple experiments, indicating improved sample quality.
• The learned latent representations exhibit reduced degeneracy, with fewer prominent and distinct dimensions capturing the data manifold efficiently.
• The ℓ₁-norm minimization of off-diagonal metric elements successfully promotes sparsity and local orthogonality in the intrinsic basis without enforcing strict orthogonality.
• The method maintains exact likelihood computation and does not rely on approximations, preserving the mathematical rigor of normalizing flows.
• The approach enables better disentanglement of data components, which can benefit downstream tasks such as out-of-distribution detection.
• The method is computationally efficient and can be applied to existing MLF frameworks with minimal modification, as demonstrated in both synthetic and image data experiments.

(b) Density plot for a fuzzy line learned with CMF, via Equation 8 .