[Paper Review] Unpacking Large Language Models with Conceptual Consistency

Pritish Kumar Sahu, Michael Cogswell+2 more|arXiv (Cornell University)|Sep 29, 2022

Topic Modeling4 citations

TL;DR

This paper introduces conceptual consistency as a metric to evaluate whether large language models (LLMs) possess coherent, background-knowledge-aligned understanding of concepts relevant to their reasoning. By extracting conceptually relevant knowledge from ConceptNet and measuring model responses via a majority-vote prompting strategy, the authors find that while conceptual consistency increases with model scale, it remains moderate and varies significantly across relations and concepts, indicating limited generalization of conceptual understanding even in large models.

ABSTRACT

If a Large Language Model (LLM) answers "yes" to the question "Are mountains tall?" then does it know what a mountain is? Can you rely on it responding correctly or incorrectly to other questions about mountains? The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply a general understanding of concepts relevant to the anchor query. We propose conceptual consistency to measure a LLM's understanding of relevant concepts. This novel metric measures how well a model can be characterized by finding out how consistent its responses to queries about conceptually relevant background knowledge are. To compute it we extract background knowledge by traversing paths between concepts in a knowledge base and then try to predict the model's response to the anchor query from the background knowledge. We investigate the performance of current LLMs in a commonsense reasoning setting using the CSQA dataset and the ConceptNet knowledge base. While conceptual consistency, like other metrics, does increase with the scale of the LLM used, we find that popular models do not necessarily have high conceptual consistency. Our analysis also shows significant variation in conceptual consistency across different kinds of relations, concepts, and prompts. This serves as a step toward building models that humans can apply a theory of mind to, and thus interact with intuitively.

Motivation & Objective

• To develop a metric that measures whether LLMs' reasoning behavior is consistent with their underlying conceptual knowledge.
• To investigate whether large language models generalize conceptual understanding beyond pattern-matching, especially in commonsense reasoning.
• To evaluate how well model performance on question-answering tasks correlates with their knowledge of conceptually relevant background facts.
• To identify systematic variations in conceptual consistency across different relations, concepts, and model scales.

Proposed method

• Extract conceptually relevant background knowledge from ConceptNet by traversing paths between concepts related to each question in the CSQA dataset.
• Construct background knowledge questions based on conceptually relevant relations (e.g., 'is a', 'desires', 'causes') to probe model knowledge.
• Apply a majority-vote-style zero-shot prompting procedure across multiple prompt variations to robustly measure model responses to background knowledge and anchor questions.
• Compute conceptual consistency as the correlation between background knowledge performance and answer correctness on the anchor task.
• Use a majority-vote prompting strategy to reduce linguistic variation bias and improve reliability of model response measurement.
• Analyze performance across three LLM families (OPT, GPT, T0) at scales up to 66B parameters to assess trends in conceptual consistency.

Experimental results

Research questions

• RQ1 To what extent does a model’s performance on commonsense reasoning tasks reflect its underlying knowledge of relevant background concepts?
• RQ2 How does conceptual consistency—defined as alignment between background knowledge and reasoning performance—vary across different types of relations and concepts?
• RQ3 Does conceptual consistency increase with model scale, and if so, how does it compare to the increase in raw knowledge or reasoning accuracy?
• RQ4 How do prompting strategies, especially majority-vote prompting, affect the measurement of background knowledge and conceptual consistency?
• RQ5 Are there specific relations or concepts where even large models show low conceptual consistency despite high performance on reasoning tasks?

Key findings

• Conceptual consistency increases with model scale but remains moderate, indicating that larger models do not necessarily achieve high alignment between background knowledge and reasoning performance.
• While knowledge of background facts increases with scale, it does so at a slower rate than reasoning accuracy or conceptual consistency, suggesting a hierarchy of skills in LLMs.
• Significant variation in conceptual consistency is observed across different relations: for example, larger models do not become more consistent about 'desires' despite improved knowledge, while they do become more consistent about 'causes'.
• Smaller models often show inconsistent behavior—being knowledgeable yet inconsistent or consistent yet ignorant—especially for relations like 'desires' and 'causes'.
• Performance varies across model families: OPT models show less robust scaling trends in conceptual consistency compared to GPT and T0, and some smaller models outperform larger ones on specific relations like 'Similar To'.
• Prompting biases (e.g., 'yes' bias in smaller OPT models) are detectable and mitigated by the majority-vote prompting strategy, which improves measurement reliability.