[Paper Review] Sum-of-Squares Polynomial Flow

Priyank Jaini, Kira A. Selby+1 more|arXiv (Cornell University)|May 7, 2019

Metabolomics and Mass Spectrometry Studies39 citations

TL;DR

The paper introduces Sum-of-Squares (SOS) flows, a universal, interpretable triangular-map-based density estimator that generalizes autoregressive flows and normalizing flows by using increasing univariate polynomials with a learnable conditioner network.

ABSTRACT

Triangular map is a recent construct in probability theory that allows one to transform any source probability density function to any target density function. Based on triangular maps, we propose a general framework for high-dimensional density estimation, by specifying one-dimensional transformations (equivalently conditional densities) and appropriate conditioner networks. This framework (a) reveals the commonalities and differences of existing autoregressive and flow based methods, (b) allows a unified understanding of the limitations and representation power of these recent approaches and, (c) motivates us to uncover a new Sum-of-Squares (SOS) flow that is interpretable, universal, and easy to train. We perform several synthetic experiments on various density geometries to demonstrate the benefits (and short-comings) of such transformations. SOS flows achieve competitive results in simulations and several real-world datasets.

Motivation & Objective

• Formulate a rigorous framework for density estimation via increasing triangular maps.
• Unify and compare autoregressive models and normalizing flows within this framework.
• Propose and analyze the Sum-of-Squares (SOS) flow as a universal, interpretable density estimator.
• Show how SOS flows generalize IAF and relate to existing flow methods.
• Demonstrate the efficacy of SOS flows through synthetic and real-world experiments.

Proposed method

• Parameterize triangular maps via one-dimensional increasing polynomials with coefficients produced by a conditioner network: T_j(z_1,...,z_j)=P_{2r+1}(z_j; a_j) where a_j=C_j(z_1,...,z_{j-1}).
• Ensure monotonicity by representing P_{2r+1} as an integral of a sums-of-squares polynomial: P_{2r+1}(z; a)=c+∫_0^z ∑_{κ=1}^k (∑_{l=0}^r a_{l,κ} u^l)^2 du.
• Prove universality: increasing polynomials are dense in the space of increasing continuous functions, enabling approximation of any increasing triangular map as r grows or via stacking blocks.
• Show SOS flows strictly generalize IAF (r=0) and provide interpretable parameters controlling higher-order moments.
• Stack multiple SOS blocks to trade depth vs width and enhance approximation capacity.
• Compare to existing autoregressive and flow-based methods and demonstrate competitive performance on synthetic and real datasets.

Experimental results

Research questions

• RQ1 Can triangular maps provide a complete and tractable framework for high-dimensional density estimation?
• RQ2 Do SOS flows offer universal approximation power for any target density while remaining interpretable and trainable?
• RQ3 How do SOS flows relate to and generalize existing autoregressive and normalizing flow methods?
• RQ4 What are the practical trade-offs (depth vs width) when implementing SOS flows for real-world data?
• RQ5 Do SOS flows perform competitively against state-of-the-art density estimators on standard benchmarks?

Key findings

• SOS flows are universal: with enough model complexity they can approximate any target density.
• SOS flows strictly generalize inverse autoregressive flow (IAF) and encompass existing autoregressive and flow-based models within the triangular-map framework.
• The polynomial-based conditional densities are monotone and computable via univariate polynomials, enabling efficient density evaluation and inversion.
• Coefficients of the polynomials directly control higher-order moments of the target density, enhancing interpretability.
• Stacking SOS blocks balances model capacity and training efficiency, with deeper vs wider configurations offering different trade-offs.
• Empirical results on synthetic and real-world datasets show SOS flows achieve competitive log-likelihoods compared to several baseline methods.