[Paper Review] HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning

Hitomi Yanaka, Koji Mineshima+5 more|arXiv (Cornell University)|Apr 27, 2019

Topic Modeling24 references4 citations

TL;DR

This paper introduces HELP, a large-scale NLI dataset designed to improve neural models' performance on monotonicity reasoning—particularly downward inferences involving lexical replacements and logical operators. By augmenting MultiNLI with carefully constructed examples, the authors show that model accuracy improves significantly on monotonicity inferences, especially those involving lexical replacements, though gains are limited for disjunction and modification in downward contexts.

ABSTRACT

Large crowdsourced datasets are widely used for training and evaluating neural models on natural language inference (NLI). Despite these efforts, neural models have a hard time capturing logical inferences, including those licensed by phrase replacements, so-called monotonicity reasoning. Since no large dataset has been developed for monotonicity reasoning, it is still unclear whether the main obstacle is the size of datasets or the model architectures themselves. To investigate this issue, we introduce a new dataset, called HELP, for handling entailments with lexical and logical phenomena. We add it to training data for the state-of-the-art neural models and evaluate them on test sets for monotonicity phenomena. The results showed that our data augmentation improved the overall accuracy. We also find that the improvement is better on monotonicity inferences with lexical replacements than on downward inferences with disjunction and modification. This suggests that some types of inferences can be improved by our data augmentation while others are immune to it.

Motivation & Objective

• To address the lack of large-scale, high-quality datasets focused on monotonicity reasoning in NLI, particularly downward inferences.
• To investigate whether the poor performance of neural models on monotonicity inferences stems from insufficient training data or architectural limitations.
• To evaluate whether data augmentation with a new dataset (HELP) can improve model generalization on logical and lexical inferences.
• To analyze which types of monotonicity inferences benefit from data augmentation and which remain challenging despite it.

Proposed method

• Developed a method to automatically generate a large-scale NLI dataset (HELP) by applying monotonicity rules to existing sentences, focusing on phrase replacements and logical operators.
• Generated 21,000 examples by systematically replacing nouns with hyponyms/hypernyms and modifying verbs with adverbs, preserving entailment relationships based on monotonicity properties.
• Augmented the MultiNLI training set with HELP data, creating a new training configuration (MultiNLI+HELP) for comparison with MultiNLI and MultiNLI+MQ.
• Trained three state-of-the-art models—BERT, BiLSTM+ELMo+Attn, and ESIM—on the augmented data and evaluated them on multiple test sets including GLUE, FraCaS, SICK, and MultiNLI.
• Used Matthews correlation coefficient (GLUE) and accuracy (other datasets) as evaluation metrics, with ablation analysis to assess performance stability on MultiNLI.
• Conducted error analysis on 68 misclassified problems from GLUE and FraCaS to identify persistent failure patterns, especially in disjunction and downward modification contexts.

Experimental results

Research questions

• RQ1 Can data augmentation with a purposefully constructed dataset (HELP) improve neural models’ performance on monotonicity reasoning in NLI?
• RQ2 Does the size of training data or the quality of reasoning patterns (e.g., lexical vs. logical) have a greater impact on model performance for monotonicity inferences?
• RQ3 Which types of monotonicity inferences (e.g., lexical replacement, disjunction, modification) benefit most from data augmentation, and which remain difficult?
• RQ4 To what extent does adding HELP data degrade performance on standard NLI benchmarks like MultiNLI?
• RQ5 Are the persistent failures in disjunction and downward modification cases due to data scarcity or inherent model limitations?

Key findings

• Adding HELP to MultiNLI improved the overall accuracy of all three tested models (BERT, BiLSTM+ELMo+Attn, ESIM) on GLUE, FraCaS, and SICK benchmarks.
• The improvement was most significant for monotonicity inferences involving lexical replacements, where models trained with HELP correctly answered 68 previously misclassified problems.
• Despite HELP being smaller than the MQ dataset (892K), it outperformed MQ in improving model accuracy, indicating that targeted data quality matters more than sheer size.
• ESIM showed the largest performance gain, supporting prior findings that tree-structured architectures may better capture logical reasoning.
• Models still struggled with downward inferences involving disjunction and modifiers, misclassifying 68 problems across GLUE and FraCaS, suggesting these are inherently difficult even with augmented data.
• The performance on MultiNLI remained stable or slightly decreased, indicating that the models learned to distinguish upward and downward monotonicity contexts without catastrophic forgetting.