[Paper Review] COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Meng Yu, Chenyan Xiong+5 more|arXiv (Cornell University)|Feb 16, 2021

Topic ModelingComputer Science70 references129 citations

TL;DR

COCO-LM pretrains language models by jointly correcting corrupted text tokens and contrasting corrupted/cropped sequences, achieving state-of-the-art GLUE/SQuAD results with improved efficiency.

ABSTRACT

We present a self-supervised learning framework, COCO-LM, that pretrains Language Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model to corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second sequence-level task, Sequence Contrastive Learning, is to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-art pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.

Motivation & Objective

• Motivate more efficient and effective pretraining beyond standard MLM/ELECTRA frameworks.
• Introduce two new pretraining tasks to address token-level semantics and sequence representations: Corrective Language Modeling (CLM) and Sequence Contrastive Learning (SCL).
• Demonstrate that COCO-LM achieves higher accuracy on GLUE and SQuAD with reduced pretraining compute.
• Show that the approach yields better token representations and alignment/uniformity in representation space.

Proposed method

• Use an auxiliary generator-like model to corrupt input sequences similar to ELECTRA.
• Train a main Transformer with: CLM to detect and correct replaced tokens; and SCL to align corrupted sequences with cropped originals.
• CLM combines a copy mechanism and language modeling loss in a multi-task setup.
• SCL applies a contrastive loss on sequence representations using paired (corrupted, cropped) positives and random negatives.
• Overall objective is L_COCO-LM = L_Aux.MLM + L_Main.CLM + L_Main.SCL.
• Provide network configurations where the auxiliary model is smaller but capable, and the main model follows standard BERT/ELECTRA architectures.

Experimental results

Research questions

• RQ1 Can correcting corrupted tokens and contrasting corrupted vs cropped sequences improve pretraining efficiency and downstream performance?
• RQ2 Do CLM and SCL provide complementary benefits to token-level semantics and sequence-level representation space?
• RQ3 How does COCO-LM compare to ELECTRA and RoBERTa in GLUE and SQuAD under equal training budgets?
• RQ4 What are the effects of data augmentation strength (crop fraction) and architectural choices on performance and generalization?

Key findings

• COCO-LM outperforms recent state-of-the-art pretraining models on GLUE average and SQuAD across base, base++, and large++ settings.
• Under same pretraining setup, COCO-LM matches MNLI accuracy of RoBERTa and ELECTRA with 60% and 50% of their GPU hours respectively.
• With the same steps, COCO-LM/Base++ outperforms previous best models by 1+ GLUE average points.
• COCO-LM/Large++ with 367M parameters reaches MNLI accuracy comparable to Megatron 3.9B.
• SCL improves alignment and uniformity of representations, reduces degeneration, and enhances few-shot generalization; CLM provides robust token-level modeling on corrupted sequences.