[Paper Review] Multi-Perspective Context Matching for Machine Comprehension

Zhiguo Wang, Haitao Mi+2 more|arXiv (Cornell University)|Dec 13, 2016

Topic Modeling19 references115 citations

TL;DR

The paper introduces the Multi-Perspective Context Matching (MPCM) model for extractive machine comprehension on SQuAD, predicting answer spans by matching passage context to the question from multiple perspectives and aggregating with BiLSTMs.

ABSTRACT

Previous machine comprehension (MC) datasets are either too small to train end-to-end deep learning models, or not difficult enough to evaluate the ability of current MC techniques. The newly released SQuAD dataset alleviates these limitations, and gives us a chance to develop more realistic MC models. Based on this dataset, we propose a Multi-Perspective Context Matching (MPCM) model, which is an end-to-end system that directly predicts the answer beginning and ending points in a passage. Our model first adjusts each word-embedding vector in the passage by multiplying a relevancy weight computed against the question. Then, we encode the question and weighted passage by using bi-directional LSTMs. For each point in the passage, our model matches the context of this point against the encoded question from multiple perspectives and produces a matching vector. Given those matched vectors, we employ another bi-directional LSTM to aggregate all the information and predict the beginning and ending points. Experimental result on the test set of SQuAD shows that our model achieves a competitive result on the leaderboard.

Motivation & Objective

• Motivate the need for realistic, large-scale MC evaluation data (SQuAD) and end-to-end modeling.
• Propose an end-to-end MPCM model that directly predicts the answer span start and end points.
• Demonstrate that multi-perspective matching improves span identification over baselines.
• Show ablation results to identify crucial components of the MPCM architecture.

Proposed method

• Word representation with fixed (GloVe) and character-composed embeddings.
• Filter layer that re-weights passage words by question relevance r_j.
• BiLSTM-based context representation for question and passage.
• Multi-Perspective Context Matching Layer with l perspectives and multiple strategies (full, maxpooling, meanpooling).
• Aggregation Layer using BiLSTM to fuse matching vectors.
• Prediction Layer with two separate softmax heads for a_b and a_e to yield Pr(a_b|Q,P) and Pr(a_e|Q,P).

Experimental results

Research questions

• RQ1 How competitive is the MPCM model on the SQuAD test set relative to contemporary methods?
• RQ2 Does the multi-perspective matching approach improve question-aware passage representations and span prediction?
• RQ3 What is the impact of the number of perspectives (l) on performance?
• RQ4 Which components of the MPCM architecture contribute most to performance (ablation analysis)?

Key findings

• Single MPCM achieves EM 65.5 and F1 75.1 on SQuAD test set.
• Ensemble MPCM improves to EM 68.2 and F1 77.2.
• MPCM compares favorably with several baselines (e.g., BiDAF EM 68.0, F1 77.3; r-net EM 69.5, F1 77.9).
• Increasing the number of perspectives from 1 to 50 yields performance gains (EM 60.7→66.1, F1 71.7→75.8).
• Ablation shows aggregation layer and certain matching strategies as crucial to performance.
• Layer ablations indicate removing any key component degrades results, with aggregation being particularly impactful.