[Paper Review] A Systematic Investigation of Commonsense Knowledge in Large Language Models

Xiang Lorraine Li, Adhiguna Kuncoro+4 more|arXiv (Cornell University)|Oct 31, 2021

Topic Modeling4 citations

TL;DR

This paper conducts a rigorous zero-shot and few-shot evaluation of large language models (up to 280B parameters) on commonsense benchmarks, controlling for surface cues and evaluation artifacts. It finds that despite strong performance, models still fall short of human-level commonsense understanding, and increasing model size or using few-shot prompting fails to close the gap without task-specific supervision.

ABSTRACT

Language models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. Here we aim to better understand the extent to which such models learn commonsense knowledge -- a critical component of many NLP applications. We conduct a systematic and rigorous zero-shot and few-shot commonsense evaluation of large pre-trained LMs, where we: (i) carefully control for the LMs' ability to exploit potential surface cues and annotation artefacts, and (ii) account for variations in performance that arise from factors that are not related to commonsense knowledge. Our findings highlight the limitations of pre-trained LMs in acquiring commonsense knowledge without task-specific supervision; furthermore, using larger models or few-shot evaluation are insufficient to achieve human-level commonsense performance.

Motivation & Objective

• To rigorously evaluate how well large language models acquire commonsense knowledge without task-specific fine-tuning.
• To isolate the extent to which model performance stems from actual commonsense reasoning versus surface cues or annotation artifacts.
• To assess the impact of evaluation design choices—such as prompt format and scoring functions—on performance fluctuations.
• To determine whether increasing model size or using few-shot in-context learning improves commonsense reasoning.
• To challenge the assumption that large language models are robustly capable of out-of-the-box commonsense reasoning.

Proposed method

• The study evaluates the Gopher language model family (up to 280B parameters) on four commonsense benchmarks in zero-shot and few-shot settings.
• A strict answer-only baseline is used to control for performance due to statistical guessing, independent of reasoning.
• Performance is evaluated across multiple prompt formats and score functions to assess robustness to evaluation design.
• Scaling behavior is analyzed by examining performance trends across six model sizes, from 125M to 280B parameters.
• The evaluation is conducted without any fine-tuning, ensuring that observed performance reflects pre-trained knowledge acquisition.
• Statistical controls are applied to isolate the effect of commonsense knowledge from spurious correlations in the data.

Experimental results

Research questions

• RQ1 To what extent do large language models truly understand commonsense knowledge, or do they rely on surface-level cues in the data?
• RQ2 How do evaluation design choices—such as prompt format and score function—influence model performance, independent of commonsense reasoning?
• RQ3 Does increasing model size or using few-shot in-context learning significantly improve commonsense reasoning in zero-shot settings?
• RQ4 Can current large language models achieve human-level performance on commonsense benchmarks without task-specific supervision?
• RQ5 What is the required model scale to reach human-level performance, assuming no explicit commonsense supervision?

Key findings

• The answer-only baseline outperforms many models on certain benchmarks, indicating that surface-level statistical patterns significantly contribute to performance.
• Even with 280B parameters, models fail to reach human-level performance on three out of four benchmarks in zero-shot settings.
• Scaling up model size alone is insufficient to close the performance gap with humans, requiring models with 100T to 10^18 parameters—far beyond current feasibility.
• Few-shot prompting improves performance only on Social IQa, with minimal gains on other benchmarks, failing to close the gap with state-of-the-art models.
• Evaluation design choices—such as prompt format and score function—can cause performance fluctuations of up to 19%, highlighting sensitivity to non-commonsense factors.
• The results suggest that current large language models cannot reliably acquire human-level commonsense knowledge without explicit supervision, multi-modal grounding, or other inductive biases.