[Paper Review] Exploring Machine Reading Comprehension with Explicit Knowledge

Chao Wang, Hui Jiang|arXiv (Cornell University)|Sep 10, 2018

Topic Modeling12 references4 citations

TL;DR

This paper proposes Knowledge Aided Reader (KAR), an MRC model that integrates explicit knowledge from WordNet-based data enrichment to improve answer span prediction. By controlling the amount of extracted semantic relations, KAR achieves an EM of 72.4 and F1 of 81.1 on SQuAD’s development set, with a 2% performance variation linked directly to knowledge quantity, proving the effectiveness of explicit knowledge in MRC.

ABSTRACT

To apply general knowledge to machine reading comprehension (MRC), we propose an innovative MRC approach, which consists of a WordNet-based data enrichment method and an MRC model named as Knowledge Aided Reader (KAR). The data enrichment method uses the semantic relations of WordNet to extract semantic level inter-word connections from each passage-question pair in the MRC dataset, and allows us to control the amount of the extraction results by setting a hyper-parameter. KAR uses the extraction results of the data enrichment method as explicit knowledge to assist the prediction of answer spans. According to the experimental results, the single model of KAR achieves an Exact Match (EM) of $72.4$ and an F1 Score of $81.1$ on the development set of SQuAD, and more importantly, by applying different settings in the data enrichment method to change the amount of the extraction results, there is a $2\%$ variation in the resulting performance of KAR, which implies that the explicit knowledge provided by the data enrichment method plays an effective role in the training of KAR.

Motivation & Objective

• To improve machine reading comprehension (MRC) by integrating explicit general knowledge into the model training process.
• To address the limitation of existing MRC models that rely solely on context without leveraging external semantic knowledge.
• To investigate how the quantity of explicitly extracted knowledge affects MRC model performance.
• To develop a data enrichment method that systematically extracts semantic relations from passage-question pairs using WordNet.

Proposed method

• A WordNet-based data enrichment method extracts semantic relations between words in passage-question pairs to generate explicit knowledge.
• The method controls the volume of extracted relations via a hyper-parameter, enabling tuning of knowledge input size.
• The Knowledge Aided Reader (KAR) model incorporates the enriched semantic relations as explicit knowledge during answer span prediction.
• KAR uses attention mechanisms to integrate the explicit knowledge with context and question representations for improved reasoning.
• The model is trained end-to-end on the SQuAD dataset with the enriched knowledge as an auxiliary input.
• Performance is evaluated using EM and F1 scores on the SQuAD development set under varying knowledge quantity settings.

Experimental results

Research questions

• RQ1 How does the integration of explicit knowledge from WordNet affect MRC model performance?
• RQ2 To what extent does the quantity of extracted knowledge influence the prediction accuracy of the MRC model?
• RQ3 Can a data enrichment method based on WordNet effectively extract semantically relevant inter-word connections from MRC data?
• RQ4 Does the use of explicit knowledge lead to consistent improvements across different knowledge quantity settings?

Key findings

• The single-model KAR achieves an Exact Match (EM) score of 72.4 and an F1 score of 81.1 on the SQuAD development set.
• Performance varies by up to 2% across different settings of the data enrichment hyper-parameter, demonstrating a direct correlation between knowledge quantity and model accuracy.
• The variation in performance confirms that the explicit knowledge extracted via WordNet plays a significant and measurable role in improving MRC predictions.
• The results indicate that controlled knowledge injection enhances model reasoning, particularly in identifying correct answer spans.
• The method successfully integrates external semantic knowledge into MRC without requiring additional training data beyond the original SQuAD annotations.
• The effectiveness of the knowledge injection is validated through consistent performance shifts with increasing knowledge input, suggesting scalability and controllability.