[Paper Review] Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators

Takeshi Teshima, Isao Ishikawa+4 more|arXiv (Cornell University)|Jun 20, 2020

Neural Networks and Applications30 references42 citations

TL;DR

The paper proves that coupling-flow invertible neural networks (CF-INNs) are universal diffeomorphism approximators, showing affine coupling suffices for broad universality and establishing an equivalence of universality classes.

ABSTRACT

Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. However, their desirable characteristics such as analytic invertibility come at the cost of restricting the functional forms. This poses a question on their representation power: are CF-INNs universal approximators for invertible functions? Without a universality, there could be a well-behaved invertible transformation that the CF-INN can never approximate, hence it would render the model class unreliable. We answer this question by showing a convenient criterion: a CF-INN is universal if its layers contain affine coupling and invertible linear functions as special cases. As its corollary, we can affirmatively resolve a previously unsolved problem: whether normalizing flow models based on affine coupling can be universal distributional approximators. In the course of proving the universality, we prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes, a theoretical insight that is of interest by itself.

Motivation & Objective

• Assess the representation power of CF-INNs for invertible transformations using a rigorous universality framework.
• Develop a criterion to determine when CF-INNs are universal based on layer designs.
• Show that affine coupling flows (ACFs) underpin universal approximation for a broad class of diffeomorphisms.
• Resolve open questions on distributional universality for affine-coupled CF-INNs.

Proposed method

• Introduce CF-INN architecture and define coupling flows and affine coupling flows.
• Establish equivalence of universal approximation properties across diffeomorphism classes using a differential geometry-based reduction.
• Prove Lp- and sup-universality results for CF-INNs containing affine coupling, via reduction to simpler coordinate-wise transformations.
• Demonstrate that H-ACF with expressive H can yield Lp-universality for S^0_c, extending to D^2 via the equivalence theorem.
• Derive corollaries linking universality to distributional universality for CF-INNs.

Experimental results

Research questions

• RQ1 Can CF-INNs attain universal approximation for broad classes of diffeomorphisms, and under what layer designs?
• RQ2 Does embedding affine coupling in CF-NN architectures guarantee universality for D^2 diffeomorphisms?
• RQ3 What is the relationship between different universality notions (Lp, sup, distributional) within CF-INN architectures?

Key findings

• CF-INNs are universal diffeomorphism approximators when their layers include affine coupling and invertible linear functions as special cases.
• An equivalence theorem shows universality for D^2, T^∞, and S^∞_c classes, enabling reduction to simpler coordinate-wise problems.
• INN_G built from H-ACF layers is an Lp-universal approximator for S^0_c, hence for D^2 via the equivalence result.
• This work confirms distributional universality for affine-coupled CF-INNs as a corollary of the Lp/universality results.
• Existing architectures like DSF and SoS are encompassed under the proven framework, inheriting their distributional universality and, in some cases, sup-universality.