[Paper Review] Transforming Question Answering Datasets Into Natural Language Inference Datasets

Dorottya Demszky, Kelvin Guu+1 more|arXiv (Cornell University)|Sep 9, 2018

Topic ModelingComputer Science32 references122 citations

TL;DR

The paper automates converting QA data into large-scale NLI datasets via a QA2D transformation, creates QA-NLI with over 500k examples, and trains a neural QA2D model that generalizes across QA domains.

ABSTRACT

Existing datasets for natural language inference (NLI) have propelled research on language understanding. We propose a new method for automatically deriving NLI datasets from the growing abundance of large-scale question answering datasets. Our approach hinges on learning a sentence transformation model which converts question-answer pairs into their declarative forms. Despite being primarily trained on a single QA dataset, we show that it can be successfully applied to a variety of other QA resources. Using this system, we automatically derive a new freely available dataset of over 500k NLI examples (QA-NLI), and show that it exhibits a wide range of inference phenomena rarely seen in previous NLI datasets.

Motivation & Objective

• Motivate diversifying NLI by leveraging abundant QA datasets.
• Propose a fully automated QA2D transformation to create entailed/non-entailed NLI pairs from QA data.
• Demonstrate that QA-derived NLI covers a wide range of inference phenomena beyond existing datasets.
• Provide a scalable pipeline combining rule-based, crowdsourced, and neural QA2D components to produce high-quality declaratives.

Proposed method

• Define QA2D: transform a QA pair (passage P, question Q, answer A) into a declarative D = A paraphrased to answer Q.
• Construct entailed NLI pairs (P, D) when A is correct; construct non-entailments when A is incorrect or Q is unanswerable.
• Develop three QA2D approaches: rule-based, crowdsourced, and neural sequence model.
• Crowdsource gold declaratives via two setups (write from scratch or post-edit rule-based outputs) to train a neural QA2D model.
• Train a neural encoder-decoder model with Q and A as inputs to generate D, using a copy mechanism and GloVe embeddings.

Experimental results

Research questions

• RQ1 Can QA-derived declaratives yield valid NLI pairs across diverse QA datasets?
• RQ2 How well does a neural QA2D model generalize to QA datasets beyond SQuAD?
• RQ3 Do automated QA2D-generated NLI datasets reveal a broader range of inference phenomena than traditional NLI corpora?
• RQ4 What is the impact of QA2D quality on NLI learning and annotation artifacts?

Key findings

• QA2D can convert QA examples into entailed non-entailed NLI pairs, enabling large-scale NLI construction.
• The neural QA2D model consistently outperforms the rule-based system in BLEU and exact-match metrics across datasets.
• The neural model achieves higher BLEU and exact-match scores (average gains: ~2.6 BLEU and ~6.2% accuracy) than the rule-based approach.
• The QA-NLI dataset, derived from five QA sources, contains over 500k NLI examples with diverse inference phenomena, including multi-sentence and meta-level reasoning.
• QA-NLI reduces some annotation artifacts observed in SNLI/MultiNLI and exhibits varied inference types across domains (movie plots, news, Wikipedia, exams).