[Paper Review] NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity

Sang-gil Lee, Sungwon Kim+1 more|arXiv (Cornell University)|Jun 11, 2020

Generative Adversarial Networks and Image Synthesis29 references4 citations

TL;DR

NanoFlow proposes a parameter-efficient normalizing flow architecture that uses a single shared neural density estimator across multiple flow stages, reducing parameter complexity to near-constant levels. By introducing parameter decomposition and flow indication embeddings, it achieves competitive density estimation and generation performance with significantly lower memory usage, enabling scalable, sublinearly complex normalizing flows.

ABSTRACT

Normalizing flows (NFs) have become a prominent method for deep generative models that allow for an analytic probability density estimation and efficient synthesis. However, a flow-based network is considered to be inefficient in parameter complexity because of reduced expressiveness of bijective mapping, which renders the models unfeasibly expensive in terms of parameters. We present an alternative parameterization scheme called NanoFlow, which uses a single neural density estimator to model multiple transformation stages. Hence, we propose an efficient parameter decomposition method and the concept of flow indication embedding, which are key missing components that enable density estimation from a single neural network. Experiments performed on audio and image models confirm that our method provides a new parameter-efficient solution for scalable NFs with significant sublinear parameter complexity.

Motivation & Objective

• To address the high parameter complexity of traditional normalizing flows, which limits scalability in real-world applications.
• To decouple model expressiveness from linearly growing parameter counts by rethinking flow parameterization.
• To enable a single neural network to model multiple flow transformations without sacrificing invertibility or expressiveness.
• To systematically evaluate the feasibility of shared density estimation across flow stacks in a parameter-efficient manner.
• To provide a modular, scalable foundation for future flow-based models with controlled trade-offs between expressiveness and memory cost.

Proposed method

• Introduces a shared neural density estimator that parameterizes all flow transformations, drastically reducing per-flow parameter overhead.
• Employs a parameter decomposition technique to enhance expressiveness of the shared representation without increasing model size.
• Proposes flow indication embeddings to condition the shared estimator on flow-specific information, enabling distinct density modeling across stages.
• Maintains invertibility by ensuring the Jacobian determinant remains analytically tractable through structured flow design.
• Uses a bipartite flow architecture to enable efficient Jacobian computation and stable training.
• Applies the shared estimator across multiple flow stacks, achieving sublinear growth in parameter count with increasing flow depth.

Experimental results

Research questions

• RQ1 Can a single neural density estimator effectively model multiple flow transformations while preserving invertibility?
• RQ2 Does parameter decomposition and flow indication embedding enable sufficient expressiveness in a shared estimator setting?
• RQ3 To what extent does shared estimation reduce parameter complexity without degrading performance?
• RQ4 How does the method perform under varying levels of autoregressive bias in the flow architecture?
• RQ5 Can the shared estimator approach scale to complex generative tasks like audio and image modeling with minimal memory overhead?

Key findings

• NanoFlow achieves sublinear parameter complexity, with model size growing nearly independently of the number of flow layers.
• The method outperforms standard normalizing flows in parameter efficiency while maintaining competitive likelihood scores on image and audio benchmarks.
• Flow indication embeddings significantly improve modeling capacity of the shared estimator, enabling accurate density estimation across multiple flow stages.
• Parameter decomposition enhances expressiveness with minimal additional parameters, demonstrating effective utilization of shared representations.
• The approach performs best on bipartite flows, suggesting a strong synergy between the architecture and the proposed parameterization scheme.
• Experiments confirm that NanoFlow enables scalable, memory-efficient normalizing flows suitable for real-world deployment.