[Paper Review] Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation

Hung Thinh Truong, Yulia Otmakhova+4 more|arXiv (Cornell University)|Oct 6, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces the NaN-NLI test suite, a linguistically grounded NLI benchmark focused on sub-clausal negation—where negation scope is ambiguous and difficult to resolve. Using a systematic framework based on Pullum and Huddleston's typology, the authors construct premise-hypothesis pairs with minimal modifications to probe model understanding, demonstrating that current pre-trained language models struggle significantly with sub-clausal and quantified negation, especially in determining logical bounds.

ABSTRACT

Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP methods, with the aim of understanding sub-clausal negation. The test suite contains premise--hypothesis pairs where the premise contains sub-clausal negation and the hypothesis is constructed by making minimal modifications to the premise in order to reflect different possible interpretations. Aside from adopting standard NLI labels, our test suite is systematically constructed under a rigorous linguistic framework. It includes annotation of negation types and constructions grounded in linguistic theory, as well as the operations used to construct hypotheses. This facilitates fine-grained analysis of model performance. We conduct experiments using pre-trained language models to demonstrate that our test suite is more challenging than existing benchmarks focused on negation, and show how our annotation supports a deeper understanding of the current NLI capabilities in terms of negation and quantification.

Motivation & Objective

• To address the lack of systematic evaluation for sub-clausal negation in NLP benchmarks, which are predominantly focused on clausal, verbal, and analytic negation.
• To develop a test suite that enables fine-grained analysis of model performance across different negation types and constructions using a formal linguistic framework.
• To investigate the interaction between negation and quantification, revealing that quantification significantly increases the difficulty of inference tasks.
• To demonstrate that existing pre-trained language models fail to correctly resolve negation scope in sub-clausal constructions, especially when combined with quantifiers.
• To provide a publicly available, annotated benchmark that supports error analysis and future research on negation and quantification in NLP.

Proposed method

• The authors construct premise-hypothesis pairs based on linguistic constructions from Pullum and Huddleston (2002), focusing on non-verbal, synthetic, and sub-clausal negation.
• Each premise is systematically modified with minimal changes to generate hypotheses reflecting different interpretations, ensuring controlled variation in negation scope.
• The test suite includes linguistic annotations for negation type (e.g., verbal/non-verbal, analytic/synthetic), construction, and transformation operations used to generate hypotheses.
• The dataset is evaluated using standard NLI labels (entailment, contradiction, neutral), with performance measured via F1 scores on both the full test set and a quantification subset (NaN-Quant).
• A gender-augmentation heuristic is applied to test for bias, comparing model predictions on original and gender-swapped pronoun/name variants.
• Experiments are conducted on pre-trained language models to compare performance on NaN-NLI against existing benchmarks, with detailed error analysis on quantification-related failures.

Figure 1: Confusion matrices of RoBERTa -MNLI on different negation-focused NLI benchmarks. C, E, N denote the Contradiction , Entailment , and Neutral class respectively.

Experimental results

Research questions

• RQ1 How do pre-trained language models perform on sub-clausal negation compared to existing NLI benchmarks?
• RQ2 To what extent do linguistic annotations (e.g., negation type, construction, operation) help explain model failures in negation resolution?
• RQ3 How does the inclusion of quantifiers affect model performance on negation tasks?
• RQ4 Does gender bias in pronouns or names affect model predictions in negation-sensitive contexts?
• RQ5 What are the specific failure patterns in model reasoning when negation scope is ambiguous or quantified?

Key findings

• The NaN-NLI test suite is significantly more challenging than existing negation-focused benchmarks, with an overall F1 score of 0.629, substantially lower than standard benchmarks.
• On the quantification subset (NaN-Quant), model performance drops further, with an F1 of 0.486, indicating that quantification exacerbates the difficulty of negation resolution.
• The model shows a marked decline in contradiction detection (F1 = 0.477 on NaN-Quant), often failing to recognize when a quantified proposition exceeds its logical bounds.
• The model incorrectly predicts Entailment when the hypothesis exceeds the bound in the premise (e.g., 'more than three' when only 'three' is stated), suggesting poor handling of quantifier bounds.
• Gender-augmented examples show no significant difference in model predictions, indicating that gender bias does not systematically affect negation interpretation in this test suite.
• Error analysis reveals that models frequently fail to detect the correct negation scope in sub-clausal constructions, especially when negation is non-verbal or synthetic.