[Paper Review] Pre-training Text-to-Text Transformers for Concept-centric Common Sense

Wangchunshu Zhou, Dong‐Ho Lee+3 more|arXiv (Cornell University)|May 3, 2021

Topic ModelingComputer Science39 references35 citations

TL;DR

This paper proposes a concept-centric pre-training framework for text-to-text transformers that integrates generative and contrastive objectives to explicitly inject commonsense knowledge into pre-trained models. By fine-tuning a T5-based model (CALM) on these intermediate tasks, the method improves performance across five commonsense benchmarks, outperforming baselines consistently in both NLU and NLG tasks without relying on external knowledge bases.

ABSTRACT

Pretrained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks that require a syntactic and semantic understanding of the text. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational and compositional commonsense knowledge about everyday concepts, which is crucial to many downstream tasks requiring commonsense reasoning. To augment PTLMs with common sense, we propose generative and contrastive objectives as intermediate self-supervised pre-training tasks between general pre-training and downstream task-specific fine-tuning. We also propose a joint training framework to unify generative and contrastive objectives so that these objectives can be more effective. Our proposed objectives can pack more commonsense knowledge into the parameters of a pre-trained text-to-text transformer without relying on external knowledge bases, yielding better performance on both NLU and NLG tasks. We apply our method on a pre-trained T5 model in an intermediate task transfer learning fashion to train a concept-aware language model (CALM) and experiment with five commonsense benchmarks (four NLU tasks and one NLG task). Experimental results show that CALM outperforms baseline methods by a consistent margin.

Motivation & Objective

• To address the limitation of existing pre-training objectives in modeling relational and compositional commonsense knowledge about everyday concepts.
• To develop intermediate self-supervised pre-training tasks that explicitly inject commonsense knowledge into text-to-text transformers.
• To unify generative and contrastive objectives in a joint training framework for more effective knowledge packing.
• To improve performance on downstream NLU and NLG tasks requiring commonsense reasoning without external knowledge bases.
• To demonstrate the effectiveness of concept-centric pre-training through extensive evaluation on five commonsense benchmarks.

Proposed method

• Proposes a generative objective that trains the model to predict commonsense concepts given a context, enhancing knowledge integration.
• Introduces a contrastive objective that encourages the model to distinguish between correct and incorrect commonsense concepts in a masked context.
• Designs a joint training framework that simultaneously optimizes both generative and contrastive objectives during intermediate pre-training.
• Applies the method to a pre-trained T5 model in a transfer learning setup, resulting in a concept-aware language model (CALM).
• Uses a curated dataset of commonsense examples to pre-train CALM on the dual objectives before fine-tuning on downstream tasks.
• Employs a text-to-text format to maintain compatibility with existing T5-style models and downstream task pipelines.

Experimental results

Research questions

• RQ1 Can joint generative and contrastive pre-training objectives effectively inject commonsense knowledge into a pre-trained text-to-text transformer?
• RQ2 How does the proposed method improve performance on commonsense reasoning tasks compared to standard pre-training objectives?
• RQ3 To what extent does the absence of external knowledge bases affect the model's performance, and can internal knowledge packing suffice?
• RQ4 Does the joint optimization of generative and contrastive objectives lead to better generalization than using either objective alone?
• RQ5 How does CALM perform across diverse commonsense benchmarks, including both NLU and NLG tasks?

Key findings

• CALM outperforms baseline models on all five commonsense benchmarks, demonstrating consistent improvements across both NLU and NLG tasks.
• The joint training of generative and contrastive objectives leads to better knowledge integration than using either objective in isolation.
• The model achieves improved performance without relying on external knowledge bases, showing that internal parameterization of commonsense knowledge is effective.
• The method successfully enhances the model's ability to reason about everyday concepts through intermediate pre-training.
• The results indicate that concept-centric pre-training significantly boosts reasoning capabilities in text-to-text transformers.
• The improvements are particularly notable in tasks requiring compositional and relational commonsense understanding.