[Paper Review] Looking Beyond Sentence-Level Natural Language Inference for Downstream Tasks

Anshuman Mishra, Dhruvesh Patel+4 more|arXiv (Cornell University)|Sep 18, 2020

Topic Modeling24 references4 citations

TL;DR

This paper proposes transforming existing multiple-choice reading comprehension (MCRC) datasets into long-premise Natural Language Inference (NLI) datasets to improve multi-sentence reasoning in NLP models. Using a hybrid conversion method that balances grammatical coherence and structural fidelity, models trained on these new datasets achieve state-of-the-art performance on downstream tasks like factual consistency checking of summaries (CFCS) and MCRC, outperforming models trained on standard sentence-level NLI data, especially on longer premises.

ABSTRACT

In recent years, the Natural Language Inference (NLI) task has garnered significant attention, with new datasets and models achieving near human-level performance on it. However, the full promise of NLI -- particularly that it learns knowledge that should be generalizable to other downstream NLP tasks -- has not been realized. In this paper, we study this unfulfilled promise from the lens of two downstream tasks: question answering (QA), and text summarization. We conjecture that a key difference between the NLI datasets and these downstream tasks concerns the length of the premise; and that creating new long premise NLI datasets out of existing QA datasets is a promising avenue for training a truly generalizable NLI model. We validate our conjecture by showing competitive results on the task of QA and obtaining the best reported results on the task of Checking Factual Correctness of Summaries.

Motivation & Objective

• To investigate why NLI models trained on sentence-level datasets fail to generalize to downstream tasks like question answering and summarization.
• To identify premise length as a key distributional gap between existing NLI datasets and real-world downstream tasks.
• To develop and evaluate conversion strategies that transform existing MCRC datasets into high-quality, long-premise NLI datasets.
• To demonstrate that models trained on these new datasets outperform those trained on standard NLI data in long-context reasoning tasks.
• To provide a new, broadly useful resource for training truly generalizable NLI models.

Proposed method

• Three conversion strategies are proposed: rule-based (heuristic sentence splitting), neural (neural sequence-to-sequence generation), and hybrid (combining rule-based structure with neural refinement).
• The hybrid method uses rule-based templates to structure premise-hypothesis pairs and applies neural models to improve grammatical coherence and fluency.
• Existing MCRC datasets (e.g., RACE, MultiRC, DREAM, CosmosQA, RACE) are repurposed into NLI-style datasets with long, multi-sentence premises.
• Models are fine-tuned on the converted NLI datasets and evaluated on two downstream tasks: MCRC (as a classification problem) and CFCS (both classification and ranking).
• Performance is analyzed across varying premise lengths to assess generalization to longer contexts.
• The quality of the converted datasets is validated by showing competitive performance on MCRC and state-of-the-art results on CFCS.

Experimental results

Research questions

• RQ1 Does the length of the premise in NLI datasets significantly affect model transferability to downstream tasks like QA and summarization?
• RQ2 Can existing MCRC datasets be effectively converted into high-quality, long-premise NLI datasets without sacrificing grammatical or structural coherence?
• RQ3 How do different conversion strategies (rule-based, neural, hybrid) compare in preserving the semantic and syntactic quality of the resulting NLI instances?
• RQ4 Do models trained on long-premise NLI datasets outperform those trained on sentence-level NLI datasets in downstream tasks requiring multi-sentence reasoning?
• RQ5 Is there a measurable performance drop for models trained on short-premise NLI data when applied to long-premise downstream tasks?

Key findings

• Models trained on the hybrid-converted RACE Hybrid dataset achieved 86.55% balanced accuracy and 0.73 F1-score on the CFCS classification task, outperforming all prior reported results.
• The model trained on RACE Hybrid, which has the longest average premise length (188.5 words), significantly outperformed models trained on shorter-premise datasets like MultiNLI, especially as premise length increased beyond 200 words.
• On the sentence-pair ranking task, the RACE Hybrid model achieved 75.70% accuracy, demonstrating strong performance on long-context inference, while the MultiNLI model dropped sharply beyond 200-word premises.
• The hybrid conversion method produced the highest-quality NLI instances, balancing grammatical coherence and structural fidelity better than rule-based or neural-only approaches.
• Performance on both MCRC and CFCS tasks declined more sharply for models trained on short-premise NLI data (e.g., MultiNLI) as premise length increased, confirming the importance of long-context training.
• The results provide strong empirical evidence that the distributional gap in premise length is a key reason for poor transfer from sentence-level NLI to downstream tasks requiring multi-sentence reasoning.