[Paper Review] Improving Generalization by Incorporating Coverage in Natural Language Inference

Nafise Sadat Moosavi, Prasetya Ajie Utama+2 more|TUbilio (Technical University of Darmstadt)|Sep 19, 2019

Topic Modeling25 references4 citations

TL;DR

This paper proposes enhancing neural network inputs in natural language inference (NLI) by incorporating coverage vectors that measure how well each word or bigram in the hypothesis is matched by the premise. By integrating these coverage representations into model inputs without external data, the approach significantly improves generalization across multiple NLI datasets—boosting performance by up to 30 points—and also enhances transfer performance on related tasks like reading comprehension and QA-SRL.

ABSTRACT

The task of natural language inference (NLI) is to identify the relation between the given premise and hypothesis. While recent NLI models achieve very high performance on individual datasets, they fail to generalize across similar datasets. This indicates that they are solving NLI datasets instead of the task itself. In order to improve generalization, we propose to extend the input representations with an abstract view of the relation between the hypothesis and the premise, i.e., how well the individual words, or word n-grams, of the hypothesis are covered by the premise. Our experiments show that the use of this information considerably improves generalization across different NLI datasets without requiring any external knowledge or additional data. Finally, we show that using the coverage information is not only beneficial for improving the performance across different datasets of the same task. The resulting generalization improves the performance across datasets that belong to similar but not the same tasks.

Motivation & Objective

• Address the poor generalization of NLI models across similar datasets despite high performance on individual benchmarks.
• Identify that current models over-rely on lexical forms and fail to learn abstract inference knowledge.
• Propose a method to improve generalization using only intrinsic, existing data features—specifically, coverage of hypothesis words and n-grams by the premise.
• Evaluate the method's transferability to related NLP tasks such as reading comprehension and QA-SRL.
• Demonstrate that coverage information enhances model robustness and performance across diverse but similar NLP tasks without requiring external knowledge or additional data.

Proposed method

• Compute a similarity matrix S = Φ(H)Φ(P)ᵀ between hypothesis and premise representations, where Φ captures word-level embeddings or hidden layer outputs.
• Derive coverage values ci = maxj(Sij) as the maximum similarity of each hypothesis word to any premise word, indicating how well it is covered.
• Extend coverage to bigrams using a CNN with window size 2 to compute representation Φ′(H), and compute c′i as the maximum similarity of each bigram to premise bigrams.
• Incorporate coverage vectors C (for words) and C′ (for bigrams), along with coverage positions Q and Q′, as additional input features to the model.
• Tune the use of C, C′, Q, Q′, and the representation layer Φ via hyperparameter search on in-domain development sets.
• Train models end-to-end with these enhanced inputs, using standard NLI and QA architectures, and evaluate on out-of-domain test sets.

Experimental results

Research questions

• RQ1 Can coverage-based representations improve generalization across different NLI datasets without external knowledge or data?
• RQ2 Does incorporating word and n-gram coverage enhance model performance on out-of-domain NLI datasets, especially when the task formulation varies?
• RQ3 Can coverage information improve generalization across related but distinct NLP tasks such as reading comprehension and QA-SRL?
• RQ4 How do coverage vectors and positions contribute to detecting non-entailment cases where lexical matches exist but semantic alignment fails?
• RQ5 Does the use of coverage improve performance on datasets with high lexical similarity in neutral pairs, such as SICK, where models trained on SNLI/MultiNLI struggle?

Key findings

• Incorporating coverage vectors improves generalization across NLI datasets by up to 30 points in accuracy, with consistent gains on SNLI, MultiNLI, and Glockner datasets.
• The model trained on SNLI or MultiNLI shows significantly improved performance on the SICK dataset when coverage is used, particularly in identifying neutral pairs with high lexical overlap.
• Coverage information improves performance on out-of-domain QA tasks: EM and F1 scores increase on the QA-SRL test set when models are trained on SQuAD with coverage features.
• The use of bigram coverage (C′) is beneficial for MQAN, while only word-level coverage (C) improves BIDAF, indicating model-specific sensitivity to representation types.
• Coverage positions (Q, Q′) help detect cases where covered words or phrases are misaligned in order, such as in H3 where the hypothesis is neutral but all words are covered out of order.
• The method improves performance across tasks without requiring additional data or external knowledge, demonstrating its effectiveness as a simple, general-purpose inductive bias for NLP models.