[Paper Review] DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models

Shansan Gong, Mukai Li+3 more|arXiv (Cornell University)|Oct 17, 2022

Topic ModelingComputer Science94 citations

TL;DR

DiffuSeq introduces a classifier-free diffusion model for Seq2Seq text generation, enabling parallel (non-autoregressive) decoding with strong quality and notable diversity, and connects diffusion to autoregressive and iterative-NAR frameworks.

ABSTRACT

Recently, diffusion models have emerged as a new paradigm for generative models. Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models to natural language is under-explored due to the discrete nature of texts, especially for conditional generation. We tackle this challenge by proposing DiffuSeq: a diffusion model designed for sequence-to-sequence (Seq2Seq) text generation tasks. Upon extensive evaluation over a wide range of Seq2Seq tasks, we find DiffuSeq achieving comparable or even better performance than six established baselines, including a state-of-the-art model that is based on pre-trained language models. Apart from quality, an intriguing property of DiffuSeq is its high diversity during generation, which is desired in many Seq2Seq tasks. We further include a theoretical analysis revealing the connection between DiffuSeq and autoregressive/non-autoregressive models. Bringing together theoretical analysis and empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks. Code is available at \url{https://github.com/Shark-NLP/DiffuSeq}

Motivation & Objective

• Motivate diffusion models for discrete, conditioned text generation in Seq2Seq tasks.
• Develop a classifier-free diffusion model that conditions on source sequences without external classifiers.
• Enable non-autoregressive, parallel decoding to improve diversity while maintaining quality.
• Establish theoretical links between DiffuSeq and AR/iter-NAR/Fully-NAR models.
• Demonstrate empirical effectiveness across multiple Seq2Seq tasks.

Proposed method

• Embed discrete text pairs (source and target) into a shared continuous space and apply a partial noising forward process that only perturbs the target portion.
• Model the reverse denoising with a Transformer-based network to learn pθ(z t−1|z t) without auxiliary classifiers (classifier-free).
• Use a unified Emb(wx ⊕ wy) embedding for joint training of source and target representations.
• Derive and minimize the variational lower bound L_VLB with a simplified objective that emphasizes y0 reconstruction and embedding consistency.
• Apply importance sampling over diffusion steps to stabilize training and employ MBR decoding to improve final quality.
• Establish connections to autoregressive, iterative-NAR, and fully-NAR models, arguing DiffuSeq extends iterative-NAR.

Experimental results

Research questions

• RQ1 Can diffusion models be effectively adapted to conditional Seq2Seq text generation without classifiers?
• RQ2 How does a partial noising forward process affect conditional generation and dependency modeling between source wx and target wy?
• RQ3 What is the relationship between DiffuSeq and AR/iter-NAR/Fully-NAR models, and does DiffuSeq offer benefits in quality and diversity?
• RQ4 Does joint training of a shared embedding for wx and wy improve performance compared to decoupled or pre-extracted representations?
• RQ5 Do diffusion-based Seq2Seq models achieve competitive quality with stronger diversity across standard Seq2Seq tasks?

Key findings

• DiffuSeq achieves comparable or superior quality to six strong baselines, including a state-of-the-art PLM-based model, across four Seq2Seq tasks.
• DiffuSeq consistently yields higher diversity (lower self-BLEU and higher div-4) while maintaining competitive BLEU, ROUGE, and BERTScore.
• The model exhibits strong sentence-level diversity and can outperform autoregressive baselines when diversity is leveraged (e.g., with larger candidate sets in MBR).
• Joint training of shared embeddings for wx and wy is important for performance; decoupled training strategies degrade results.
• DiffuSeq provides a theoretical and empirical bridge between autoregressive, iterative-NAR, and diffusion approaches, establishing diffusion as a viable extension for conditional language generation.