[Paper Review] e-SNLI: Natural Language Inference with Natural Language Explanations

Oana-Maria Camburu, Tim Rocktäschel+2 more|arXiv (Cornell University)|Dec 4, 2018

Topic Modeling282 citations

TL;DR

The paper extends SNLI with natural language explanations (e-SNLI) and shows how explanations can be used to train models, generate justifications, improve universal sentence representations, and transfer to out-of-domain NLI datasets.

ABSTRACT

In order for machine learning to garner widespread public adoption, models must be able to provide interpretable and robust explanations for their decisions, as well as learn from human-provided explanations at train time. In this work, we extend the Stanford Natural Language Inference dataset with an additional layer of human-annotated natural language explanations of the entailment relations. We further implement models that incorporate these explanations into their training process and output them at test time. We show how our corpus of explanations, which we call e-SNLI, can be used for various goals, such as obtaining full sentence justifications of a model's decisions, improving universal sentence representations and transferring to out-of-domain NLI datasets. Our dataset thus opens up a range of research directions for using natural language explanations, both for improving models and for asserting their trust.

Motivation & Objective

• Motivate the use of free-form natural language explanations to improve model interpretability and robustness.
• Create a large corpus of explanations (e-SNLI) aligned with SNLI labels.
• Investigate how explanations can be used during training to enhance sentence representations and transfer to new domains.

Proposed method

• Collect a large, crowd-sourced corpus of explanations for SNLI premises–hypotheses pairs using constrained prompts and two-step annotation.
• Integrate explanations into neural models by extending the InferSent framework with an explanation decoder.
• Train models to predict labels and generate explanations (PredictAndExplain) and to generate explanations first and then predict labels (ExplainThenPredict).
• Evaluate whether explanations improve universal sentence representations via transfer tasks (SentEval).
• Assess cross-domain transfer performance to SICK-E and MultiNLI without fine-tuning.

Experimental results

Research questions

• RQ1 Can free-form natural language explanations be reliably collected for SNLI pairs?
• RQ2 Do explanations provide additional supervisory signals that improve label prediction and sentence representations?
• RQ3 Can models generate coherent explanations that justify their predictions?
• RQ4 Do explanations help models transfer better to out-of-domain NLI datasets?

Key findings

• Explanation-aware models can generate justifications conditioned on predicted labels without sacrificing label accuracy (e-InferSent achieves 83.96% test accuracy versus 84.01% for the baseline).
• The explanation decoder can produce reasonable explanations with perplexity 10.58 and BLEU 22.40 on the test set for the PredictAndExplain setup.
• ExplainThenPredict with attention yields higher-quality explanations (64.27% correct explanations on a manual subset) at the cost of some drop in label accuracy (81.71%).
• Training with explanations improves universal sentence representations on downstream tasks compared to the InferSent baseline in several metrics (e- InferSent shows significant gains on multiple tasks).
• Direct transfer without fine-tuning to SICK-E and MultiNLI shows small accuracy changes, with explanations providing qualitative insights into model behavior rather than large gains in out-of-domain label accuracy.