[Paper Review] Unification-based Reconstruction of Explanations for Science Questions.

Marco Valentino, Mokanarangan Thayaparan+1 more|arXiv (Cornell University)|Mar 31, 2020

Topic Modeling4 citations

TL;DR

This paper proposes a unification-based framework that reconstructs explanations for science questions by combining relevance and unification scores to rank explanatory facts. It achieves state-of-the-art performance on the Worldtree corpus, improving BERT accuracy by up to 15.69% on challenging multiple-choice questions while remaining scalable to large knowledge bases.

ABSTRACT

The paper presents a framework to reconstruct explanations for multiple choices science questions through explanation-centred corpora. Building upon the notion of unification in science, the framework ranks explanatory facts with respect to question and candidate answer by leveraging a combination of two different scores: (a) A Relevance Score (RS) that represents the extent to which a given fact is specific to the question; (b) A Unification Score (US) that takes into account the explanatory power of a fact, determined according to its frequency in explanations for similar questions. An extensive evaluation of the framework is performed on the Worldtree corpus, adopting IR weighting schemes for its implementation. The following findings are presented: (1) The proposed approach achieves competitive results when compared to state-of-the-art Transformers, yet possessing the property of being scalable to large explanatory knowledge bases; (2) The combined model significantly outperforms IR baselines (+7.8/8.4 MAP), confirming the complementary aspects of relevance and unification score; (3) The constructed explanations can support downstream models for answer prediction, improving the accuracy of BERT for multiple choices QA on both ARC easy (+6.92%) and challenge (+15.69%) questions.

Motivation & Objective

• To develop a scalable method for reconstructing explanations for science questions using explanatory knowledge bases.
• To address the challenge of combining fact relevance and explanatory power in multi-choice science QA.
• To improve downstream answer prediction by integrating reconstructed explanations into neural models.
• To evaluate the framework on a large-scale explanation-centered corpus (Worldtree) using IR weighting schemes.

Proposed method

• The framework uses a Relevance Score (RS) to measure how specific a fact is to the question and candidate answer.
• It computes a Unification Score (US) based on a fact's frequency in explanations for similar questions, capturing its explanatory power.
• The two scores are combined to rank explanatory facts, with higher scores indicating stronger explanatory relevance.
• The model is implemented using IR weighting schemes for efficient computation and scalability.
• The framework is evaluated on the Worldtree corpus, using MAP as the primary metric for performance comparison.

Experimental results

Research questions

• RQ1 Can a unification-based approach effectively reconstruct explanations for science questions using explanatory corpora?
• RQ2 How do relevance and unification scores complement each other in identifying high-quality explanatory facts?
• RQ3 To what extent can reconstructed explanations improve answer prediction in multiple-choice science QA?
• RQ4 How does the framework compare to state-of-the-art Transformer models in terms of accuracy and scalability?

Key findings

• The framework achieves competitive performance compared to state-of-the-art Transformers while being scalable to large explanatory knowledge bases.
• The combined model outperforms IR baselines by +7.8/8.4 MAP, confirming the complementary value of relevance and unification scores.
• Reconstructed explanations improve BERT’s accuracy on ARC easy questions by 6.92% and on challenge questions by 15.69%.
• The Unification Score effectively captures explanatory power through frequency patterns in similar question explanations.