[Paper Review] A Survey on Explainability in Machine Reading Comprehension

Mokanarangan Thayaparan, Marco Valentino+1 more|arXiv (Cornell University)|Oct 1, 2020

Topic Modeling117 references32 citations

TL;DR

This paper systematically reviews benchmarks and architectures for explainability in Machine Reading Comprehension (MRC), detailing knowledge-based and operational explanations, benchmarks, evaluation metrics, and open research questions.

ABSTRACT

This paper presents a systematic review of benchmarks and approaches for explainability in Machine Reading Comprehension (MRC). We present how the representation and inference challenges evolved and the steps which were taken to tackle these challenges. We also present the evaluation methodologies to assess the performance of explainable systems. In addition, we identify persisting open research questions and highlight critical directions for future work.

Motivation & Objective

• Define explainability in the context of MRC and motivate its importance for evaluation, generalisation, and interpretability.
• Catalogue explainability benchmarks and classify them by task domain, format, MRC type, and explanation characteristics.
• Classify architectural approaches to explainable MRC into knowledge-based, operational, and hybrid categories, and analyze their learning paradigms.
• Discuss evaluation metrics for explainability and identify challenges in benchmarking explainable MRC.
• Outline open research questions and future directions for advancing explainable MRC.

Proposed method

• Systematic review of explainability in MRC from 2015 onwards across AI/NLP venues.
• Classification of benchmarks along dimensions such as domain, format, MRC type, multi-hop, explanation type, and representation.
• Taxonomy of architectural patterns for explanation generation, including knowledge-based, latent/neural, and hybrid models.
• Analysis of evaluation methodologies and the concept of silver explanations for training.
• Discussion of open research problems including contrastive explanations and faithfulness of explanations.

Experimental results

Research questions

• RQ1 What are the main benchmarks and datasets used to develop and evaluate explainable MRC models?
• RQ2 What architectural patterns are used to generate and integrate explanations in MRC, and how do they differ across extractive vs abstractive tasks?
• RQ3 How is explainability evaluated in MRC, and what metrics and challenges exist in benchmarking explanations?
• RQ4 What are the limitations and open questions in current explainable MRC research, and what directions are recommended for future work?

Key findings

• Explainability in MRC is pursued via two main explanation types: knowledge-based explanations and operational explanations, with a growing emphasis on abstractive reasoning.
• Benchmarks for explainable MRC increasingly incorporate multi-hop reasoning and explicit explanation signals, including both extractive and abstractive tasks across open-domain, science, and commonsense domains.
• Architectural trends show a shift toward supervised neural approaches for explainability, with increasing use of transformers and graph networks to model explanatory relevance.
• Evaluation of explainability combines exact matching, F1, ranking metrics, and language-generation metrics, with concerns about the faithfulness and quality of explanations.
• Recent work introduces dataset and methodological innovations such as question decomposition, explicit inference chains, and neuro-symbolic approaches to improve explainability and generalisation.
• The paper identifies open questions, including the need for contrastive explanations and improved faithfulness of explanations.