[論文レビュー] KERMIT: Generative Insertion-Based Modeling for Sequences

William Chan, Nikita Kitaev+3 more|arXiv (Cornell University)|Jun 4, 2019

Natural Language Processing Techniques参考文献 22被引用数 67

ひとこと要約

KERMITは、固定された因子分解を用ず、p(x,y)とその周辺分布/条件付きを共同学習する統一の挿入ベースモデルを提示し、双方向翻訳、表現学習、ゼロショットクローズQAを対数時間の並列デコードで実現します。

ABSTRACT

We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x, y)$ to learn the joint distribution $p(x, y)$, and optionally mix in unpaired data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we have access to the conditionals $p(x \mid y)$ and $p(y \mid x)$ in both directions. We can also sample from the joint distribution or the marginals. The model supports both serial fully autoregressive decoding and parallel partially autoregressive decoding, with the latter exhibiting an empirically logarithmic runtime. We demonstrate through experiments in machine translation, representation learning, and zero-shot cloze question answering that our unified approach is capable of matching or exceeding the performance of dedicated state-of-the-art systems across a wide range of tasks without the need for problem-specific architectural adaptation.

研究の動機と目的

• Motivate a flexible sequence modeling framework that does not rely on a prespecified left-to-right factorization.
• Learn a joint distribution over sequences and its marginals/conditionals in a unified model.
• Enable bidirectional generation and infilling, including translation and cloze-style QA.
• Demonstrate competitive performance across machine translation, representation learning, and zero-shot QA using a simple Transformer-based architecture.

提案手法

• Model sequences via insertion operations that build a canvas in any order to represent the joint distribution p(x,y).
• Train by lower-bounding the log-likelihood with Jensen’s inequality, sampling generation orders and insertions.
• Factorize the content and location as p(c,l)=p(c|l)p(l) and use a single Transformer decoder without causal masking.
• Enable inference in both directions (p(y|x) and p(x|y)) and sampling from the joint and marginals.
• Extend to pairs of sequences by concatenating x and y and training to learn joint, marginal, and conditional decompositions.

実験結果

リサーチクエスチョン

• RQ1 Can an insertion-based model learn the joint distribution p(x,y) and its decompositions without a fixed factorization?
• RQ2 Does a single unified model match or exceed state-of-the-art performance on translation, representation learning, and cloze QA?
• RQ3 How does bidirectional generation and marginal refinement affect performance and efficiency compared to traditional autoregressive models?
• RQ4 What are the inference and sampling capabilities when modeling pairs of sequences with insertion operations?

主な発見

• KERMIT can match or exceed state-of-the-art performance on machine translation, representation learning, and zero-shot cloze QA across tasks.
• The model supports both serial autoregressive decoding and parallel partially autoregressive decoding with empirically logarithmic runtime in sequence length.
• Joint modeling with marginal refining (p(x) and p(y)) improves translation quality in German→English by about 1.2 BLEU in the reported setup.
• Bidirectional training and finetuning provide competitive results without problem-specific architectural adaptations.
• Insertion-based decoding enables dynamic growth of the output canvas, avoiding fixed-length generation constraints.