[Paper Review] Stochastic Answer Networks for Natural Language Inference

Xiaodong Liu, Kevin Duh+1 more|arXiv (Cornell University)|Apr 21, 2018

Topic Modeling24 references51 citations

TL;DR

The paper introduces Stochastic Answer Networks (SAN) that perform multi-step inference for natural language inference, refining a hidden state across passes and achieving state-of-the-art results on SNLI, MultiNLI, SciTail, and Quora Question Pairs.

ABSTRACT

We propose a stochastic answer network (SAN) to explore multi-step inference strategies in Natural Language Inference. Rather than directly predicting the results given the inputs, the model maintains a state and iteratively refines its predictions. Our experiments show that SAN achieves the state-of-the-art results on three benchmarks: Stanford Natural Language Inference (SNLI) dataset, MultiGenre Natural Language Inference (MultiNLI) dataset and Quora Question Pairs dataset.

Motivation & Objective

• Motivate multi-step, iterative reasoning for natural language inference (NLI) beyond single-step predictions.
• Propose a Stochastic Answer Network (SAN) architecture that maintains and updates a state across inference steps.
• Demonstrate that SAN improves over single-step baselines and achieves state-of-the-art results on multiple benchmarks.
• Show robustness and analyze the effect of the number of inference steps and stochastic prediction dropout.

Proposed method

• Four-layer SAN architecture: lexicon encoding, contextual encoding, memory generation, and an iterative answer module.
• Lexicon layer concatenates word and character embeddings to form E^p and E^h for premise and hypothesis.
• Contextual encoding layer uses two BiLSTM layers with a maxout to produce C^p and C^h.
• Memory layer builds working memory with dot-product attention and BiLSTMs to produce M^p and M^h.
• Answer module performs multi-step state updates s_t with a GRU, derives step-wise predictions P_t^r, and averages them to obtain final P^r.
• Training employs stochastic prediction dropout at the final averaging stage to encourage robustness.

Experimental results

Research questions

• RQ1 Does multi-step inference in SAN improve NLI over single-step baselines?
• RQ2 How does SAN perform across standard NLI benchmarks (SNLI, MultiNLI, SciTail, Quora)?
• RQ3 What is the optimal number of inference steps (T) for SAN, and how does it affect performance?
• RQ4 How does incorporating stochastic prediction dropout affect model robustness and performance?
• RQ5 Can the SAN approach generalize when combined with pretrained contextual embeddings or multi-task learning?

Key findings

• SAN consistently outperforms the single-step baseline on all four datasets (SNLI, MultiNLI, SciTail, Quora).
• On SciTail, SAN improves by a notable margin over the single-step model (+3.89 on dev: 85.46 vs 89.35).
• SAN achieves state-of-the-art results on SciTail and Quora Question Pairs test sets, and remains competitive on SNLI and MultiNLI without large external knowledge or pretrained embeddings.
• Using a 5-step inference configuration yielded the best results on SciTail dev and comparable benchmarks, while more steps did not consistently improve performance.