[Paper Review] Cubic-Spline Flows

Conor Durkan, Artur Bekasov+2 more|arXiv (Cornell University)|Jun 5, 2019

Advanced Numerical Analysis TechniquesEngineering20 citations

TL;DR

This paper introduces cubic-spline flows, a normalizing flow architecture that uses monotonic cubic splines as coupling transforms to achieve both fast invertibility and high density estimation performance. By stacking these splines with LU-decomposed linear layers, the model matches or exceeds autoregressive flows in density estimation while maintaining exact one-pass inversion for efficient sampling and likelihood evaluation.

ABSTRACT

A normalizing flow models a complex probability density as an invertible transformation of a simple density. The invertibility means that we can evaluate densities and generate samples from a flow. In practice, autoregressive flow-based models are slow to invert, making either density estimation or sample generation slow. Flows based on coupling transforms are fast for both tasks, but have previously performed less well at density estimation than autoregressive flows. We stack a new coupling transform, based on monotonic cubic splines, with LU-decomposed linear layers. The resulting cubic-spline flow retains an exact one-pass inverse, can be used to generate high-quality images, and closes the gap with autoregressive flows on a suite of density-estimation tasks.

Motivation & Objective

• Address the trade-off between invertibility speed and density estimation performance in normalizing flows.
• Develop a coupling transform that maintains fast one-pass inversion like Real NVP but achieves performance closer to autoregressive flows.
• Enable flexible, monotonic, and tractable density estimation using cubic splines as a drop-in replacement for affine or additive coupling layers.
• Close the performance gap between coupling-based and autoregressive flows in density estimation tasks without sacrificing sampling efficiency.

Proposed method

• Propose a monotonic cubic spline coupling transform that parameterizes the transformation using piecewise cubic polynomials with continuous derivatives.
• Use Steffen's method to enforce monotonicity by computing derivatives at knots based on local slopes and constraints, ensuring invertibility.
• Implement the Jacobian determinant using the derivative of the cubic spline, which is a piecewise quadratic function, enabling tractable log-likelihood computation.
• Integrate the cubic-spline coupling layers with LU-decomposed linear layers to enable cross-dimensional interactions and improve representational capacity.
• Enable exact inversion via analytical root-finding of cubic polynomials, ensuring one-pass inverse computation without numerical optimization.
• Train the model end-to-end using backpropagation to maximize log-likelihood on training data, leveraging the tractable Jacobian determinant.

Experimental results

Research questions

• RQ1 Can a coupling-based normalizing flow achieve density estimation performance comparable to autoregressive flows while maintaining fast one-pass inversion?
• RQ2 How does the use of monotonic cubic splines compare to previous spline-based transforms (e.g., quadratic splines) in terms of flexibility and expressivity?
• RQ3 To what extent do cubic-spline flows match or exceed state-of-the-art results in density estimation across standard benchmarks?
• RQ4 Can cubic-spline flows generate high-quality images with only a single forward pass, demonstrating practical generative capability?

Key findings

• Cubic-spline flows close the performance gap with autoregressive flows on density estimation tasks, achieving state-of-the-art results on multiple benchmarks.
• The model maintains exact one-pass inversion, enabling efficient sampling and likelihood evaluation, unlike autoregressive flows which are slow to invert.
• The use of monotonic cubic splines allows for flexible, continuous, and monotonic transformations that can interpolate arbitrary quantiles at knot locations.
• The model generates high-quality images with only a single neural network pass, demonstrating strong generative capability.
• The Jacobian determinant is efficiently computed via the derivative of the cubic spline, which is a piecewise quadratic function.
• Analytical inversion via root-finding of cubic polynomials ensures numerical stability and exactness, avoiding iterative optimization.