[Paper Review] Discrete Flows: Invertible Generative Models of Discrete Data

Dustin Tran, Keyon Vafa+3 more|arXiv (Cornell University)|May 24, 2019

Natural Language Processing Techniques40 references40 citations

TL;DR

The paper extends normalizing flows to discrete data with discrete autoregressive and discrete bipartite flows, enabling bidirectional exact likelihoods and fast nonautoregressive generation.

ABSTRACT

While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We consider two flow architectures: discrete autoregressive flows that enable bidirectionality, allowing, for example, tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows that enable efficient non-autoregressive generation as in RealNVP. Empirically, we find that discrete autoregressive flows outperform autoregressive baselines on synthetic discrete distributions, an addition task, and Potts models; and bipartite flows can obtain competitive performance with autoregressive baselines on character-level language modeling for Penn Tree Bank and text8.

Motivation & Objective

• Extend normalizing flows to discrete distributions using a discrete change-of-variables framework.
• Provide two invertible architectures for discrete data: discrete autoregressive flows and discrete bipartite flows.
• Demonstrate improved modeling and generation speed across synthetic tasks and character-level language modeling.

Proposed method

• Define a discrete change-of-variables formula p(y)=p(x=f^{-1}(y)) for invertible discrete f.
• Develop discrete autoregressive flows to stack autoregressive levels and enable bidirectionality with exact likelihoods.
• Develop discrete bipartite (coupling) flows to enable fast, parallel generation with exact likelihoods.
• Train discrete flows using straight-through gradient estimates with Gumbel-softmax relaxations for discrete outputs (temperature tau=0.1).
• Evaluate on discretized mixtures, full-rank discrete distributions, an addition task, Potts models, and character-level language modeling.

Experimental results

Research questions

• RQ1 Can normalizing flows be extended to discrete data without log-determinant-Jacobian computations?
• RQ2 Do discrete autoregressive flows enable bidirectional exact likelihoods and improve over autoregressive bases?
• RQ3 Do discrete bipartite flows provide competitive performance with faster, nonautoregressive generation?
• RQ4 How do discrete flows perform on synthetic discrete distributions and real text data (character-level language modeling)?

Key findings

• Discrete flows extend normalizing flows to discrete data via a discrete change-of-variables formula that omits log-determinant terms.
• Discrete autoregressive flows yield improved log-likelihoods over autoregressive baselines on synthetic discrete distributions, an addition task, and Potts models.
• Discrete bipartite flows achieve competitive performance with autoregressive baselines while enabling fast, parallel generation.
• Character-level language modeling with discrete bipartite flows achieves 1.38 bits per character on Penn Tree Bank and 1.23 bits per character on text8, with generation speeds over 100x faster than autoregressive models.