[Paper Review] InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Model

Sidi Lu, Tao Meng+1 more|arXiv (Cornell University)|Feb 12, 2021

Topic Modeling4 citations

TL;DR

InsNet is a novel insertion-based text generation model that enables efficient, flexible, and high-performing sequence generation by introducing an insertion-oriented relative positional encoding (offset) and a lightweight slot representation strategy, reducing training re-encoding to a single pass. It achieves state-of-the-art performance in training speed, inference efficiency, and generation quality across lexically constrained text generation and machine translation tasks, with a new decoding algorithm (InsNet-Dinic) enabling controllable parallelization.

ABSTRACT

We propose InsNet, an expressive insertion-based text generator with efficient training and flexible decoding (parallel or sequential). Unlike most existing insertion-based text generation works that require re-encoding of the context after each insertion operation and thus are inefficient to train, InsNet only requires one pass of context encoding for the entire sequence during training by introducing a novel insertion-oriented position encoding and a light-weighted slot representation strategy to enable computation sharing. Furthermore, we propose an algorithm InsNet-Dinic to better determine the parallelization of insertion operations that provides a controllable switch between parallel and sequential decoding, making it flexible to handle more parallelizable tasks such as machine translation with efficient decoding, or less parallelizable tasks such as open-domain text generation to guarantee high-quality outputs. Experiments on two lexically constrained text generation datasets and three machine translation datasets demonstrate InsNet's advantages over previous insertion-based methods in terms of training speed, inference efficiency, and generation quality.

Motivation & Objective

• Address the high training cost of insertion-based text generation models that require repeated context re-encoding after each insertion.
• Enable both parallel and sequential decoding in a unified framework to support diverse generation tasks, from structured translation to open-ended generation.
• Design a relative positional encoding scheme that maintains model expressiveness while allowing computation sharing across insertion steps.
• Develop an algorithm (InsNet-Dinic) that optimizes insertion parallelization to minimize likelihood discrepancy and improve decoding quality.
• Demonstrate superior performance in training efficiency, inference speed, and generation quality compared to prior insertion-based models.

Proposed method

• Introduce an insertion-oriented relative positional encoding called 'offset' that encodes pairwise spatial relationships between tokens, enabling fixed position encodings that do not require re-encoding during training.
• Propose 'offset compression' to efficiently compute the offset matrix for any insertion order, allowing dynamic and scalable positional encoding without recomputing context representations.
• Design a global slot representation aggregation mechanism that enables parallel computation of expressive slot representations across all insertion positions in a sequence.
• Introduce InsNet-Dinic, a decoding algorithm inspired by Dinic’s algorithm, to determine optimal insertion order parallelization levels based on insertion dependency graphs.
• Use a single-pass context encoding during training by maintaining static position encodings and updating only new token positions via the offset mechanism.
• Integrate knowledge distillation and mixed-precision training to further improve inference efficiency and model performance.

Experimental results

Research questions

• RQ1 Can insertion-based text generation models achieve single-pass context encoding during training without sacrificing model capacity or positional expressiveness?
• RQ2 How can insertion-based models support both parallel and sequential decoding while maintaining high generation quality across diverse tasks?
• RQ3 What is the impact of relative positional encoding on model performance and training efficiency in insertion-based generation?
• RQ4 Can a controllable parallelization strategy reduce likelihood discrepancy and improve decoding quality in insertion-based models?
• RQ5 How does the proposed architecture compare to existing insertion-based models in terms of training speed, inference efficiency, and generation quality?

Key findings

• InsNet reduces training re-encoding to a single pass (O(1)) by using an offset-based relative positional encoding, significantly improving training efficiency compared to prior models that require O(n) re-encoding steps.
• InsNet-Dinic enables a controllable trade-off between parallel and sequential decoding, achieving a parallelization level of approximately 15.8 insertions per step on average, which improves inference speed while maintaining high quality.
• On the Yelp and Cornell lexically constrained text generation datasets, InsNet achieves BLEU scores of 43.71 (with KD) and 44.10 (with KD), outperforming previous methods including Levenshtein Transformer and Insertion Transformer.
• InsNet trains in just 1h12min per epoch on a single RTX 3090 GPU, significantly faster than baselines such as Levenshtein Transformer (16h33min) and Insertion Transformer (8h24min).
• Ablation studies show that the offset matrix and global representation are critical: removing the global representation leads to worse convergence and higher termination NLL, while truncating the offset to [-1,1] reduces model capacity and performance.
• The model achieves state-of-the-art performance in both training speed and generation quality, with inference time of 103ms per sequence using the KD-optimized version, outperforming baselines in both speed and BLEU scores.