[Paper Review] Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey

Prajjwal Bhargava, Vincent Ng|arXiv (Cornell University)|Jan 28, 2022

Topic Modeling9 citations

TL;DR

This survey examines how pre-trained language models (PLMs) capture and utilize commonsense knowledge for reasoning and generation tasks, analyzing their strengths, limitations, and biases. It evaluates benchmarks, probes PLM knowledge through templates, and identifies key challenges like reporting bias, knowledge sparsity, and multilingual gaps, proposing future research directions for robust, human-like understanding in NLP.

ABSTRACT

While commonsense knowledge acquisition and reasoning has traditionally been a core research topic in the knowledge representation and reasoning community, recent years have seen a surge of interest in the natural language processing community in developing pre-trained models and testing their ability to address a variety of newly designed commonsense knowledge reasoning and generation tasks. This paper presents a survey of these tasks, discusses the strengths and weaknesses of state-of-the-art pre-trained models for commonsense reasoning and generation as revealed by these tasks, and reflects on future research directions.

Motivation & Objective

• To survey recent advances in commonsense knowledge reasoning and generation using pre-trained language models (PLMs).
• To analyze the extent to which PLMs capture and exploit commonsense knowledge, as revealed by recent benchmarks.
• To identify key limitations in current benchmarks, including biases and reporting bias, that hinder meaningful evaluation of model understanding.
• To explore open challenges such as knowledge sparsity, non-contextualization in knowledge graphs, and the need for multilingual commonsense reasoning.
• To guide future research by identifying underexplored components of commonsense knowledge, such as mental and emotional states, and the integration of multimodal signals.

Proposed method

• Using probing methods with hand-crafted templates to convert knowledge base triples (e.g., <subject, relation, object>) into natural language sentences for evaluating PLM knowledge.
• Employing self-supervised pre-training on large-scale text to learn universal language representations, including linguistic and commonsense knowledge.
• Applying masked language modeling (MLM) and other self-supervised tasks during pre-training to enable PLMs to learn contextual and commonsense knowledge.
• Using adversarial filtering and data augmentation to reduce lexical overlap and annotation artifacts that create shortcut learning paths in benchmarks.
• Exploring methods to densify and contextualize knowledge graphs (KGs) by leveraging structural and semantic connections to reduce sparsity and improve relevance.
• Investigating multimodal approaches that combine visual and textual signals to enhance commonsense reasoning beyond text-only training.

Experimental results

Research questions

• RQ1 To what extent do pre-trained language models capture and utilize commonsense knowledge, and how can this be measured reliably?
• RQ2 What are the primary limitations of current benchmarks in evaluating commonsense reasoning, and how do biases affect model performance and interpretation?
• RQ3 How can reporting bias and knowledge sparsity in knowledge graphs be mitigated to improve the robustness of commonsense reasoning systems?
• RQ4 What role can multimodal data (e.g., vision) play in enriching commonsense knowledge acquisition and reasoning in PLMs?
• RQ5 What are the challenges and opportunities in extending commonsense reasoning to multilingual settings, and why do cross-lingual PLMs underperform on non-English benchmarks?

Key findings

• Pre-trained language models (PLMs) demonstrate significant capability in commonsense reasoning and generation, but their performance gains on benchmarks may not reflect true understanding due to task design flaws.
• Many existing benchmarks suffer from lexical overlap, predictable question structures, and annotation artifacts, enabling models to achieve high accuracy without genuine reasoning.
• Adversarial filtering and data augmentation techniques reduce shortcut learning, but biases remain a persistent challenge in benchmark design and evaluation.
• Knowledge graphs used in commonsense reasoning are often sparse and non-contextual, limiting their utility; efforts to densify and contextualize them show promise but remain incomplete.
• Reporting bias—where commonsense knowledge is implied rather than stated—leads to over-generalization and amplification of model biases, especially with limited training data.
• Cross-lingual PLMs exhibit poor transfer performance to non-English commonsense reasoning tasks, highlighting a critical gap in multilingual commonsense understanding research.