[Paper Review] Tell Me Why Is It So? Explaining Knowledge Graph Relationships by Finding Descriptive Support Passages

Sumit Bhatia, Purusharth Dwivedi+1 more|arXiv (Cornell University)|Mar 17, 2018

Topic Modeling29 references3 citations

TL;DR

This paper proposes a probabilistic, unsupervised method that ranks passages from a large corpus (Wikipedia) to provide descriptive explanations for relationships in a knowledge graph (Wikidata). By combining passage-level and document-level evidence using language models, the approach outperforms baselines in user studies, significantly improving explainability and user trust in automated knowledge bases.

ABSTRACT

We address the problem of finding descriptive explanations of facts stored in a knowledge graph. This is important in high-risk domains such as healthcare, intelligence, etc. where users need additional information for decision making and is especially crucial for applications that rely on automatically constructed knowledge bases where machine learned systems extract facts from an input corpus and working of the extractors is opaque to the end-user. We follow an approach inspired from information retrieval and propose a simple and efficient, yet effective solution that takes into account passage level as well as document level properties to produce a ranked list of passages describing a given input relation. We test our approach using Wikidata as the knowledge base and Wikipedia as the source corpus and report results of user studies conducted to study the effectiveness of our proposed model.

Motivation & Objective

• To address the need for descriptive, human-readable explanations of knowledge graph relationships in high-stakes domains like healthcare and intelligence.
• To improve user trust and decision-making by providing contextual support for facts extracted from unstructured text.
• To develop an interpretable, unsupervised solution that does not rely on templates or pre-defined paths.
• To evaluate the effectiveness of passage-based explanations through user studies comparing relevance and informativeness.
• To enable better exploratory search and knowledge discovery by linking abstract triples to natural language context.

Proposed method

• The method uses a probabilistic ranking model that combines passage-level and document-level evidence to score candidate passages.
• Passage-level relevance is modeled using a language model that estimates the likelihood of the relationship triple appearing in a passage.
• Document-level evidence is incorporated via a boosting term that increases scores for passages from documents with high-frequency mentions of the entities involved.
• The final score is computed using a weighted combination of passage likelihood and document evidence, formalized in Equation 5 of the paper.
• The approach is unsupervised and does not require annotated training data or templates.
• The system retrieves and ranks passages from Wikipedia using Wikidata as the knowledge base, with no reliance on external knowledge or pre-trained models.

Experimental results

Research questions

• RQ1 Can passage-level and document-level evidence be effectively combined to rank explanatory passages for knowledge graph relationships?
• RQ2 How does the proposed method compare to baseline approaches that rely solely on word overlap or frequency?
• RQ3 To what extent do user studies confirm that the proposed passages improve understanding and trust in knowledge graph facts?
• RQ4 Can the method identify relevant, context-rich explanations even when key terms appear only once in a passage?
• RQ5 What types of errors or misrankings are most common in the explanation generation process?

Key findings

• The proposed method significantly outperformed a strong baseline that relied on word frequency and passage overlap in user studies.
• Passages generated by the model were rated as more informative and relevant than those from the baseline, particularly in cases where key terms appeared only once.
• The document-level evidence component played a crucial role in boosting relevant passages, even when they contained only a single mention of the target term.
• User evaluators frequently rejected passages that mentioned the entities but failed to describe the specific relationship, highlighting the importance of contextual relevance.
• Error analysis revealed two main failure modes: passages discussing entities without explaining the relationship, and passages with only indirect or fleeting references to the relationship.
• The results demonstrate that combining language modeling with document-level evidence leads to more effective and trustworthy explanations for knowledge graph facts.