[Paper Review] Max-Affine Spline Insights into Deep Generative Networks

Randall Balestriero, Sébastien Paris+1 more|arXiv (Cornell University)|Feb 26, 2020

Generative Adversarial Networks and Image Synthesis60 references4 citations

TL;DR

This paper establishes a theoretical framework linking deep generative networks (DGNs) to max-affine spline operators (MASOs), revealing how piecewise-affine manifolds shape generated data. It derives analytical expressions for the output density and entropy, enabling likelihood computation and exposing limitations in modeling low-entropy or multimodal distributions, with implications for disentanglement and training stability.

ABSTRACT

We connect a large class of Generative Deep Networks (GDNs) with spline operators in order to derive their properties, limitations, and new opportunities. By characterizing the latent space partition, dimension and angularity of the generated manifold, we relate the manifold dimension and approximation error to the sample size. The manifold-per-region affine subspace defines a local coordinate basis; we provide necessary and sufficient conditions relating those basis vectors with disentanglement. We also derive the output probability density mapped onto the generated manifold in terms of the latent space density, which enables the computation of key statistics such as its Shannon entropy. This finding also enables the computation of the GDN likelihood, which provides a new mechanism for model comparison as well as providing a quality measure for (generated) samples under the learned distribution. We demonstrate how low entropy and/or multimodal distributions are not naturally modeled by DGNs and are a cause of training instabilities.

Motivation & Objective

• To develop a theoretical framework connecting deep generative networks (DGNs) to max-affine spline operators (MASOs) for better structural understanding.
• To characterize the intrinsic dimension, angular structure, and manifold partition of generated data in DGNs.
• To derive analytical expressions for the output probability density and Shannon entropy on the generated manifold.
• To establish necessary and sufficient conditions for disentanglement in terms of local affine basis vectors.
• To reveal limitations of DGNs in modeling low-entropy or multimodal distributions and link these to training instabilities.

Proposed method

• Models DGNs as compositions of max-affine spline operators (MASOs), which are continuous, piecewise-affine (CPA) functions.
• Represents each layer as a pointwise maximum of R affine transformations: $\text{MASO}(\bm{u};\{\bm{A}_r,\bm{b}_r\}_{r=1}^R) = \max_r \bm{A}_r\bm{u} + \bm{b}_r$.
• Analyzes the latent space partition into polyhedral regions, where each region corresponds to a specific active affine map.
• Derives the Jacobian determinant of the generator map to compute the output density from the latent density via change of variables.
• Uses the local affine basis vectors in each region to define a coordinate system and analyze disentanglement via angular constraints.
• Applies the framework to analyze dropout and dropconnect, showing they induce mixture models over generator dimensions.

Experimental results

Research questions

• RQ1 How does the piecewise-affine structure of DGNs affect the intrinsic dimension and geometry of the generated manifold?
• RQ2 What conditions on the local affine basis vectors ensure disentanglement in the latent space?
• RQ3 How can the output probability density and Shannon entropy be analytically computed on the generated manifold?
• RQ4 Why do low-entropy or multimodal distributions cause training instabilities in DGNs?
• RQ5 How do dropout and weight-sharing constrain the curvature of the generated manifold despite overparameterization?

Key findings

• The intrinsic dimension of the generated manifold is determined by the number of active affine regions and their angular relationships, with dimensionality bounded by the latent space size.
• The paper derives an analytical expression for the output probability density on the generated manifold, enabling likelihood computation and model comparison.
• Shannon entropy of the output density provides a new metric for evaluating the difficulty of generating complex, low-entropy distributions with DGNs.
• Low-entropy and multimodal distributions are poorly modeled by standard DGNs, leading to training instabilities due to mismatched manifold geometry.
• Weight sharing in DGNs severely constrains the curvature of the generated manifold, even when the network is highly overparameterized.
• Dropout and dropconnect both result in a mixture of DGNs with varying dimensions and parameters, but dropconnect is less likely to reduce generator dimension due to per-entry noise application.