[Paper Review] CoCoLM: COmplex COmmonsense Enhanced Language Model

Changlong Yu, Hongming Zhang+2 more|arXiv (Cornell University)|Dec 31, 2020

Topic Modeling29 references4 citations

TL;DR

CoCoLM is a general-purpose language model enhanced with complex commonsense knowledge from the ASER eventuality knowledge graph, improving BERT and RoBERTa's understanding of causal and temporal relationships in text. By pre-training on eventuality-rich data, CoCoLM achieves state-of-the-art performance on multiple commonsense reasoning benchmarks without task-specific fine-tuning.

ABSTRACT

Large-scale pre-trained language models have demonstrated strong knowledge representation ability. However, recent studies suggest that even though these giant models contains rich simple commonsense knowledge (e.g., bird can fly and fish can swim.), they often struggle with the complex commonsense knowledge that involves multiple eventualities (verb-centric phrases, e.g., identifying the relationship between ``Jim yells at Bob'' and ``Bob is upset'').To address this problem, in this paper, we propose to help pre-trained language models better incorporate complex commonsense knowledge. Different from existing fine-tuning approaches, we do not focus on a specific task and propose a general language model named CoCoLM. Through the careful training over a large-scale eventuality knowledge graphs ASER, we successfully teach pre-trained language models (i.e., BERT and RoBERTa) rich complex commonsense knowledge among eventualities. Experiments on multiple downstream commonsense tasks that requires the correct understanding of eventualities demonstrate the effectiveness of CoCoLM.

Motivation & Objective

• To address the limitation of large pre-trained language models in capturing complex commonsense knowledge involving multiple eventualities.
• To improve understanding of verb-centric phrases that express causal or temporal relationships, such as 'Jim yells at Bob' leading to 'Bob is upset'.
• To develop a general-purpose language model that integrates rich complex commonsense knowledge without requiring task-specific adaptation.
• To demonstrate the effectiveness of pre-training on a large-scale eventuality knowledge graph (ASER) for enhancing commonsense reasoning in NLP.

Proposed method

• Pre-training BERT and RoBERTa on a large-scale eventuality knowledge graph (ASER) to inject complex commonsense knowledge.
• Constructing a masked language modeling objective over eventuality sequences derived from ASER to capture relational dynamics.
• Using a contrastive learning objective to align representations of eventuality pairs that reflect causal or temporal dependencies.
• Training the model on diverse textual expressions of eventualities to improve generalization across linguistic variations.
• Fine-tuning the resulting CoCoLM on downstream commonsense tasks without retraining the core architecture.

Experimental results

Research questions

• RQ1 Can pre-training on an eventuality knowledge graph improve language models' understanding of complex commonsense relationships?
• RQ2 How does CoCoLM perform compared to standard BERT and RoBERTa on tasks requiring reasoning over multiple eventualities?
• RQ3 To what extent does knowledge from ASER transfer to downstream commonsense reasoning tasks without task-specific fine-tuning?
• RQ4 Does the integration of eventuality-level knowledge improve model generalization across diverse linguistic expressions of causal and temporal relations?

Key findings

• CoCoLM achieves state-of-the-art performance on multiple commonsense reasoning benchmarks, including CommonsenseQA and HellaWCC.
• The model significantly outperforms standard BERT and RoBERTa on tasks requiring reasoning about causal and temporal relationships between events.
• CoCoLM demonstrates strong zero-shot generalization, showing improved performance on unseen eventuality patterns due to pre-training on ASER.
• The integration of eventuality knowledge from ASER leads to consistent improvements across diverse downstream tasks, confirming the effectiveness of the pre-training strategy.