[Paper Review] Machine Reading, Fast and Slow: When Do Models "Understand" Language?

Sagnik Ray Choudhury, Anna Rogers+1 more|arXiv (Cornell University)|Sep 15, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a framework to assess whether reading comprehension models 'understand' language by defining expected reasoning steps for coreference resolution and comparison, then evaluating BERT-family models using saliency maps and counterfactual explanations. It finds that larger models better rely on correct information for comparison, but still learn lexical patterns rather than generalizable reasoning, failing on out-of-distribution examples.

ABSTRACT

Two of the most fundamental challenges in Natural Language Understanding (NLU) at present are: (a) how to establish whether deep learning-based models score highly on NLU benchmarks for the 'right' reasons; and (b) to understand what those reasons would even be. We investigate the behavior of reading comprehension models with respect to two linguistic 'skills': coreference resolution and comparison. We propose a definition for the reasoning steps expected from a system that would be 'reading slowly', and compare that with the behavior of five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. We find that for comparison (but not coreference) the systems based on larger encoders are more likely to rely on the 'right' information, but even they struggle with generalization, suggesting that they still learn specific lexical patterns rather than the general principles of comparison.

Motivation & Objective

• To address the challenge of determining whether NLP models achieve high performance on NLU benchmarks for the 'right reasons'.
• To define what constitutes valid reasoning steps for linguistic skills like coreference resolution and comparison in reading comprehension.
• To evaluate whether BERT-based models rely on the correct information using explainability techniques.
• To investigate whether model performance generalizes beyond training data, especially under counterfactual perturbations.
• To develop a methodological framework for assessing model 'understanding' that is applicable to a broad class of models and datasets.

Proposed method

• Define expected reasoning steps for two linguistic skills: coreference resolution and comparison, using a structured, step-by-step framework.
• Use saliency-based explanations to identify which input tokens models attend to when answering questions.
• Generate human-interpretable counterfactual perturbations that change model predictions, testing robustness to semantic shifts.
• Compare saliency maps and counterfactual explanations to validate whether models rely on the correct linguistic cues.
• Evaluate five BERT-family models of varying sizes on the SQuAD dataset to assess performance and reasoning fidelity.
• Apply a behavioral test based on Def. 1 to determine if models are 'right for the right reasons' by measuring alignment between expected and actual model behavior.

Experimental results

Research questions

• RQ1 What constitutes valid reasoning for coreference resolution and comparison in reading comprehension tasks?
• RQ2 Do BERT-based models rely on the correct information when answering questions involving coreference or comparison?
• RQ3 How does model size affect the likelihood of relying on correct reasoning steps?
• RQ4 To what extent do models generalize to out-of-distribution counterfactual examples?
• RQ5 How well do saliency and counterfactual explanations align in identifying model reasoning pathways?

Key findings

• For comparison tasks, larger BERT-based models are more likely to rely on the correct information, indicating improved reasoning fidelity with scale.
• Even the largest models fail to generalize to out-of-distribution counterfactual perturbations, suggesting reliance on specific lexical patterns rather than general principles.
• For coreference resolution, no model consistently relies on the correct information, indicating persistent failure to learn the underlying linguistic phenomenon.
• Saliency maps and counterfactual explanations show partial correlation, but neither alone is sufficient to fully validate reasoning behavior.
• Smaller models may use heuristics or non-interpretable strategies, as their predictions do not align with human reasoning even when explanations are shown.
• The results suggest that current models do not learn general reasoning skills but memorize superficial patterns, limiting their trustworthiness and robustness.