[Paper Review] Bio-SODA: Enabling Natural Language Question Answering over Knowledge Graphs without Training Data

Ana Claudia Sima, Tarcisio Mendes de Farias+5 more|arXiv (Cornell University)|Apr 28, 2021

Semantic Web and Ontologies33 references16 citations

TL;DR

Bio-SODA is a training-free natural language question answering system for knowledge graphs that generates SPARQL queries using a graph-based matching approach and a novel ranking algorithm incorporating node centrality. It achieves a 20% higher F1-score than state-of-the-art systems on complex scientific KGs, including bioinformatics and EU project datasets, without requiring labeled question-answer pairs.

ABSTRACT

The problem of natural language processing over structured data has become a growing research field, both within the relational database and the Semantic Web community, with significant efforts involved in question answering over knowledge graphs (KGQA). However, many of these approaches are either specifically targeted at open-domain question answering using DBpedia, or require large training datasets to translate a natural language question to SPARQL in order to query the knowledge graph. Hence, these approaches often cannot be applied directly to complex scientific datasets where no prior training data is available. In this paper, we focus on the challenges of natural language processing over knowledge graphs of scientific datasets. In particular, we introduce Bio-SODA, a natural language processing engine that does not require training data in the form of question-answer pairs for generating SPARQL queries. Bio-SODA uses a generic graph-based approach for translating user questions to a ranked list of SPARQL candidate queries. Furthermore, Bio-SODA uses a novel ranking algorithm that includes node centrality as a measure of relevance for selecting the best SPARQL candidate query. Our experiments with real-world datasets across several scientific domains, including the official bioinformatics Question Answering over Linked Data (QALD) challenge, show that Bio-SODA outperforms publicly available KGQA systems by an F1-score of least 20% and by an even higher factor on more complex bioinformatics datasets.

Motivation & Objective

• To address the lack of training data in scientific knowledge graphs, which limits the applicability of existing neural KGQA systems.
• To enable accurate, zero-shot question answering over complex, real-world scientific knowledge graphs without requiring labeled question-answer pairs.
• To improve SPARQL query generation by combining string similarity, semantic similarity, and node centrality for better candidate ranking.
• To support generalization across diverse scientific domains with minimal manual configuration and high extensibility.
• To enhance explainability by guiding users through concept matching, query construction, and result interpretation.

Proposed method

• Bio-SODA uses a graph-based approach to match natural language questions to knowledge graph entities and relations, constructing candidate SPARQL queries from these matches.
• It applies a hybrid similarity measure combining string-level and semantic similarity between question terms and KG elements.
• A novel ranking algorithm incorporates node centrality (e.g., PageRank) as a relevance signal to prioritize semantically plausible query candidates.
• The system supports configurable search over specific properties (e.g., labels, descriptions), excluding verbose or redundant fields like full abstracts.
• It handles complex queries with multiple triple patterns (up to 10 or more) without requiring user guidance or pre-trained models.
• The pipeline includes entity linking and query graph construction, with results projected using meaningful variable names for explainability.

Experimental results

Research questions

• RQ1 Can a KGQA system achieve high performance on complex scientific knowledge graphs without relying on labeled training data?
• RQ2 How effective is node centrality as a relevance signal in ranking SPARQL query candidates for natural language questions?
• RQ3 To what extent can a graph-based, non-neural approach outperform existing neural KGQA systems on real-world scientific datasets?
• RQ4 How does the system handle ambiguity and redundancy in scientific KGs with imprecise or verbose labels?
• RQ5 Can the system be generalized across diverse scientific domains with minimal configuration?

Key findings

• Bio-SODA achieves a minimum 20% improvement in F1-score over publicly available KGQA systems on real-world scientific knowledge graphs, with even greater gains on complex bioinformatics datasets.
• The integration of node centrality into the ranking function significantly improves result quality, especially in datasets with label redundancy or imprecise terminology.
• The system successfully handles complex queries with up to 10 or more triple patterns, demonstrating robustness beyond simple two- or three-triple queries.
• On the QALD-4 and CORDIS datasets, Bio-SODA outperforms existing open-source systems, including those based on deep learning, despite lacking any training data.
• The system exhibits strong generalization across domains, including biomedical, gene orthology, and EU-funded project knowledge graphs, with minimal configuration.
• Limitations include incorrect handling of ASK queries (returning results instead of boolean answers), which the authors identify as a future extension with a question type classifier.