[Paper Review] Syntax-driven Iterative Expansion Language Models for Controllable Text Generation

Noé Casas, José A. R. Fonollosa+1 more|arXiv (Cornell University)|Apr 5, 2020

Topic Modeling30 references4 citations

TL;DR

This paper proposes Syntax-driven Iterative Expansion Language Models (IE-LMs), a non-autoregressive text generation framework that uses dependency parse trees to guide parallel, syntactically informed token generation. By decoding tokens according to the tree structure, the method achieves 55% faster inference than autoregressive models (45% of steps), maintains high fluency and diversity comparable to Transformers, and enables direct syntactic control for stylistic variation.

ABSTRACT

The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations.

Motivation & Objective

• Address the inefficiency of autoregressive text generation, which scales linearly with sequence length.
• Introduce a non-autoregressive generation paradigm that leverages syntactic structure to enable parallel decoding.
• Enable direct control over syntactic constructions in generated text for stylistic variation.
• Maintain high generation quality and diversity while significantly reducing decoding steps.
• Develop a training method compatible with batch processing and subword-level tokenization.

Proposed method

• Train a Transformer-based language model to generate tokens in an order dictated by the dependency parse tree, starting from the root and expanding branches iteratively.
• Use dependency placeholders as special tokens to represent non-terminal nodes in the parse tree during training and decoding.
• Apply tree binarization to transform the dependency tree into a binary structure, enabling efficient parallel decoding across branches.
• Condition the model on the dependency structure so that each token generation step is informed by its syntactic parent and siblings.
• Use a standard cross-entropy loss during training, enabling end-to-end, fully supervised learning without reinforcement learning.
• Enable subword-level generation by integrating subword tokenization into the dependency tree structure during decoding.

Experimental results

Research questions

• RQ1 Can dependency parse trees be used to drive a non-autoregressive, parallel decoding process that maintains high fluency and coherence?
• RQ2 To what extent can syntactic structure reduce decoding steps while preserving generation quality compared to autoregressive models?
• RQ3 Can the iterative expansion mechanism enable direct control over syntactic constructions for inducing stylistic variations?
• RQ4 How does the model perform in terms of diversity and fluency when using subword-level tokenization?
• RQ5 Does the tree-based decoding order lead to better or comparable performance to autoregressive Transformers and LSTMs in both automatic and human evaluation?

Key findings

• The iterative expansion LM reduces decoding steps to 45% of those required by autoregressive models on average, achieving a 55% speedup.
• Generated text maintains high fluency and syntactic agreement, even when tokens are generated in non-sequential, tree-ordered fashion.
• Generation quality is comparable to both LSTM and Transformer models, as confirmed by both automatic metrics and human evaluation.
• Diversity of generated text is comparable to that of autoregressive Transformers, indicating no significant degradation in output variety.
• The method supports subword-level tokenization and batch training, enabling practical deployment with modern NLP pipelines.
• Direct syntactic control is enabled through the tree structure, allowing for the induction of stylistic variations in generated text.