[Paper Review] On the Potential of Artificial Intelligence Chatbots for Data Exploration of Federated Bioinformatics Knowledge Graphs

Ana-Claudia Sima, Tarcisio Mendes de Farias|arXiv (Cornell University)|Apr 20, 2023

FinTech, Crowdfunding, Digital Finance4 citations

TL;DR

This paper explores the use of large language model chatbots like ChatGPT to facilitate natural language-based data exploration in federated bioinformatics knowledge graphs. It demonstrates how such models can summarize datasets, explain SPARQL queries, and generate federated queries from natural language, though with notable limitations including hallucinations and inconsistent reasoning.

ABSTRACT

In this paper, we present work in progress on the role of artificial intelligence (AI) chatbots, such as ChatGPT, in facilitating data access to federated knowledge graphs. In particular, we provide examples from the field of bioinformatics, to illustrate the potential use of Conversational AI to describe datasets, as well as generate and explain (federated) queries across datasets for the benefit of domain experts.

Motivation & Objective

• To investigate whether conversational AI models can improve access to federated bioinformatics knowledge graphs by translating natural language questions into SPARQL queries.
• To assess the role of AI chatbots in summarizing dataset contents to enhance data findability and reusability for domain experts.
• To evaluate the ability of chatbots to explain and generate correct federated SPARQL queries across multiple public bioinformatics databases.
• To identify key limitations of current LLM-based approaches, particularly hallucinations and lack of reproducibility, in scientific data exploration contexts.

Proposed method

• Utilized the GPT-4-based ChatGPT (March 2023 free research version) to generate descriptions, explanations, and SPARQL queries from natural language inputs.
• Focused on three public bioinformatics knowledge graphs: UniProt, OMA, and Bgee, all of which support SPARQL endpoints and are part of the SIB’s federated data ecosystem.
• Evaluated chatbot outputs by comparing them against official documentation and known correct answers from authoritative sources like NCBI and EBI.
• Assessed query generation by testing the correctness and performance of generated federated SPARQL queries on real endpoints.
• Explored the impact of input phrasing and context on output consistency, including non-deterministic responses across repeated queries.
• Investigated the reliability of model explanations by asking 'how' questions to trace the origin of factual claims.

Experimental results

Research questions

• RQ1 Can AI chatbots like ChatGPT effectively summarize the content of public bioinformatics knowledge graphs in a way that enhances data findability?
• RQ2 To what extent can chatbots accurately explain existing SPARQL queries, especially in the context of federated querying?
• RQ3 Can chatbots generate correct and efficient federated SPARQL queries from natural language questions across multiple bioinformatics knowledge graphs?
• RQ4 How do hallucinations and inconsistent reasoning affect the reliability of chatbot-generated answers in scientific data exploration?
• RQ5 What are the practical limitations of using LLM-based chatbots for scientific data access, particularly regarding reproducibility and query performance?

Key findings

• ChatGPT successfully generated high-level, understandable summaries of UniProt, OMA, and Bgee databases, improving data findability through natural language access.
• The model provided plausible explanations for existing SPARQL queries, helping users understand complex federated query structures.
• ChatGPT generated syntactically correct federated SPARQL queries from natural language questions, demonstrating potential for direct data access without manual query writing.
• Despite correct syntax, some generated queries were semantically incorrect or inefficient, leading to potential server timeouts due to poor query planning.
• The model exhibited hallucinations, such as falsely claiming the taxonomic ID of *Colocasia esculenta* was 4472, despite citing the correct source (NCBI) in its explanation.
• Explanations for answers were often inconsistent with the final output, and repeated queries sometimes produced different results, highlighting non-determinism and lack of debugging transparency.