[Paper Review] Meet in the Middle: A New Pre-training Paradigm

Anh‐Tu Nguyen, Nikos Karampatziakis+1 more|arXiv (Cornell University)|Mar 13, 2023

Topic Modeling4 citations

TL;DR

This paper proposes 'Meet in the Middle' (MIM), a novel pre-training paradigm that jointly trains a left-to-right and a right-to-left language model on the same data, enforcing agreement between their predictions to improve data efficiency and infilling performance. The method achieves state-of-the-art results in perplexity and code completion, with a bidirectional inference procedure that reduces latency and improves accuracy by leveraging context from both sides.

ABSTRACT

Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion, assuming that the next token only depends on the preceding ones. However, this assumption ignores the potential benefits of using the full sequence information during training, and the possibility of having context from both sides during inference. In this paper, we propose a new pre-training paradigm with techniques that jointly improve the training data efficiency and the capabilities of the LMs in the infilling task. The first is a training objective that aligns the predictions of a left-to-right LM with those of a right-to-left LM, trained on the same data but in reverse order. The second is a bidirectional inference procedure that enables both LMs to meet in the middle. We show the effectiveness of our pre-training paradigm with extensive experiments on both programming and natural language models, outperforming strong baselines.

Motivation & Objective

• To improve pre-training data efficiency by leveraging both prefix and suffix context during training, despite the autoregressive nature of standard language models.
• To enhance performance on infilling tasks—where users insert content into existing sequences—by enabling models to use context from both directions.
• To develop a unified training and inference framework that maintains compatibility with existing autoregressive models while improving quality and efficiency.
• To reduce inference latency in infilling tasks by detecting convergence between forward and backward predictions.

Proposed method

• Train two shared-parameter language models: one left-to-right (forward) and one right-to-left (backward), on the same monotonically ordered training data.
• Introduce a total variation distance-based agreement regularizer that aligns the predicted probability distributions of the forward and backward models at each token position.
• Use a combined loss function consisting of standard language modeling loss and the agreement regularizer to jointly optimize both models during pre-training.
• During inference, run forward and backward generation in parallel, stopping when the predicted tokens meet in the middle and agree.
• Leverage the convergence of both models as a stopping criterion, reducing the number of generation steps and improving latency.
• Use the forward model as a drop-in replacement for standard autoregressive LMs, while the backward model supports bidirectional tasks like infilling.

Experimental results

Research questions

• RQ1 Can jointly training left-to-right and right-to-left language models improve data efficiency during pre-training?
• RQ2 Does enforcing agreement between forward and backward predictions lead to better performance on downstream infilling tasks?
• RQ3 Can a bidirectional inference procedure that meets in the middle reduce latency while improving accuracy compared to autoregressive baselines?
• RQ4 How does the MIM paradigm compare to existing methods like FIM and CM3 in terms of perplexity and code completion performance?

Key findings

• The MIM pre-training paradigm achieves lower perplexity than strong baselines, including FIM, on both natural language and code datasets.
• On the HumanEval code completion benchmark, MIM outperforms FIM and other baselines in zero-shot code completion accuracy.
• The bidirectional inference procedure reduces average inference latency by up to 50% compared to FIM, particularly when the models converge early.
• Ablation studies confirm that the agreement regularizer significantly improves consistency and performance, especially in infilling tasks.
• The method maintains compatibility with existing autoregressive models, as the forward model is directly usable as a replacement for standard LMs.