[Paper Review] Knowledge-Augmented Language Models for Cause-Effect Relation Classification

Pedram Hosseini, David Broniatowski+1 more|arXiv (Cornell University)|Dec 16, 2021

Topic Modeling4 citations

TL;DR

This paper proposes a knowledge-augmented pretraining framework that enhances BERT and RoBERTa for cause-effect relation classification by continually pretraining them on automatically verbalized commonsense knowledge from ATOMIC²⁰ and GLUCOSE. The method significantly improves performance on COPA, BCOPA-CE, and TCR benchmarks without architectural changes or quality-enhanced fine-tuning data, with RoBERTa-Large achieving 85.7% accuracy on COPA-test.

ABSTRACT

Previous studies have shown the efficacy of knowledge augmentation methods in pretrained language models. However, these methods behave differently across domains and downstream tasks. In this work, we investigate the augmentation of pretrained language models with commonsense knowledge in the cause-effect relation classification and commonsense causal reasoning tasks. After automatically verbalizing ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, and GLUCOSE, a dataset of implicit commonsense causal knowledge, we continually pretrain BERT and RoBERTa with the verbalized data. Then we evaluate the resulting models on cause-effect pair classification and answering commonsense causal reasoning questions. Our results show that continually pretrained language models augmented with commonsense knowledge outperform our baselines on two commonsense causal reasoning benchmarks, COPA and BCOPA-CE, and the Temporal and Causal Reasoning (TCR) dataset, without additional improvement in model architecture or using quality-enhanced data for fine-tuning.

Motivation & Objective

• To investigate the impact of commonsense knowledge injection on cause-effect relation classification in pretrained language models.
• To evaluate whether continual pretraining with automatically verbalized knowledge improves performance on causal reasoning benchmarks.
• To demonstrate that knowledge augmentation can boost performance without architectural modifications or high-quality fine-tuning data.
• To release code and models for reproducibility and community use.

Proposed method

• Verbalize triples from the ATOMIC²⁰ knowledge graph using human-readable templates to generate natural language statements.
• Convert GLUCOSE's implicit causal knowledge into natural language using modified causal connective templates.
• Continually pretrain BERT and RoBERTa using masked language modeling on the verbalized commonsense knowledge.
• Use the resulting models for zero-shot or few-shot classification on cause-effect pair datasets.
• Evaluate performance on COPA, BCOPA-CE, and TCR benchmarks without additional fine-tuning or loss function modifications.
• Apply prompt engineering (e.g., 'It is because' or 'As a result,') to improve performance on BCOPA-CE when baseline performance is near random.

Experimental results

Research questions

• RQ1 Can continual pretraining with automatically verbalized commonsense knowledge improve cause-effect relation classification in PLMs?
• RQ2 How does knowledge augmentation from ATOMIC²⁰ and GLUCOSE compare to standard fine-tuning on causal reasoning benchmarks?
• RQ3 Does the performance gain persist on hard examples and implicit causal reasoning tasks?
• RQ4 Can smaller models like BERT-Large achieve competitive results with knowledge-augmented pretraining compared to larger models like T5-11B?
• RQ5 Does prompt engineering further enhance performance on ambiguous or implicit causal reasoning tasks?

Key findings

• RoBERTa-Large fine-tuned on GLUCOSE verbalized data achieved 85.7% accuracy on COPA-test, outperforming the baseline RoBERTa-Large (74.1%) and surpassing larger models like T5-11B (94.8%) in accuracy despite having fewer parameters.
• The GLUCOSE-augmented RoBERTa-Large model achieved 83.3% accuracy on the hard split of COPA, significantly outperforming the baseline RoBERTa-Large (70.2%) and demonstrating robustness on challenging examples.
• The ATOMIC²⁰-augmented RoBERTa-Large model achieved 84.9% accuracy on COPA when fine-tuned with only event-related relations, showing that even a subset of commonsense knowledge improves performance.
• On BCOPA-CE, the GLUCOSE-augmented RoBERTa-Large model achieved 66.1% accuracy with prompt engineering, outperforming Han and Wang’s (2021) b-l-aug and b-l-reg models.
• The knowledge-augmented models outperformed baselines on the TCR dataset without any architectural changes or additional fine-tuning data, confirming the effectiveness of continual pretraining with commonsense knowledge.
• The authors released code and models, enabling reproducibility and further research in knowledge-augmented causal reasoning.