[Paper Review] KERMIT: Generative Insertion-Based Modeling for Sequences

William Chan, Nikita Kitaev+3 more|arXiv (Cornell University)|Jun 4, 2019

Natural Language Processing Techniques22 references67 citations

TL;DR

KERMIT presents a unified insertion-based model that jointly learns p(x,y) and its marginals/conditionals without fixed factorization, enabling bidirectional translation, representation learning, and zero-shot cloze QA with logarithmic-time parallel decoding.

ABSTRACT

We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x, y)$ to learn the joint distribution $p(x, y)$, and optionally mix in unpaired data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we have access to the conditionals $p(x \mid y)$ and $p(y \mid x)$ in both directions. We can also sample from the joint distribution or the marginals. The model supports both serial fully autoregressive decoding and parallel partially autoregressive decoding, with the latter exhibiting an empirically logarithmic runtime. We demonstrate through experiments in machine translation, representation learning, and zero-shot cloze question answering that our unified approach is capable of matching or exceeding the performance of dedicated state-of-the-art systems across a wide range of tasks without the need for problem-specific architectural adaptation.

Motivation & Objective

• Motivate a flexible sequence modeling framework that does not rely on a prespecified left-to-right factorization.
• Learn a joint distribution over sequences and its marginals/conditionals in a unified model.
• Enable bidirectional generation and infilling, including translation and cloze-style QA.
• Demonstrate competitive performance across machine translation, representation learning, and zero-shot QA using a simple Transformer-based architecture.

Proposed method

• Model sequences via insertion operations that build a canvas in any order to represent the joint distribution p(x,y).
• Train by lower-bounding the log-likelihood with Jensen’s inequality, sampling generation orders and insertions.
• Factorize the content and location as p(c,l)=p(c|l)p(l) and use a single Transformer decoder without causal masking.
• Enable inference in both directions (p(y|x) and p(x|y)) and sampling from the joint and marginals.
• Extend to pairs of sequences by concatenating x and y and training to learn joint, marginal, and conditional decompositions.

Experimental results

Research questions

• RQ1 Can an insertion-based model learn the joint distribution p(x,y) and its decompositions without a fixed factorization?
• RQ2 Does a single unified model match or exceed state-of-the-art performance on translation, representation learning, and cloze QA?
• RQ3 How does bidirectional generation and marginal refinement affect performance and efficiency compared to traditional autoregressive models?
• RQ4 What are the inference and sampling capabilities when modeling pairs of sequences with insertion operations?

Key findings

• KERMIT can match or exceed state-of-the-art performance on machine translation, representation learning, and zero-shot cloze QA across tasks.
• The model supports both serial autoregressive decoding and parallel partially autoregressive decoding with empirically logarithmic runtime in sequence length.
• Joint modeling with marginal refining (p(x) and p(y)) improves translation quality in German→English by about 1.2 BLEU in the reported setup.
• Bidirectional training and finetuning provide competitive results without problem-specific architectural adaptations.
• Insertion-based decoding enables dynamic growth of the output canvas, avoiding fixed-length generation constraints.