[Paper Review] ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots

Reham Omar, Omij Mangukiya+2 more|arXiv (Cornell University)|Feb 8, 2023

Topic Modeling38 citations

TL;DR

The paper compares ChatGPT and Galactica (conversational LMs) withKGQAn and EDGQA (KGQA systems) across four real knowledge graphs, proposing a framework and outlining open research directions for KG chatbots.

ABSTRACT

Conversational AI and Question-Answering systems (QASs) for knowledge graphs (KGs) are both emerging research areas: they empower users with natural language interfaces for extracting information easily and effectively. Conversational AI simulates conversations with humans; however, it is limited by the data captured in the training datasets. In contrast, QASs retrieve the most recent information from a KG by understanding and translating the natural language question into a formal query supported by the database engine. In this paper, we present a comprehensive study of the characteristics of the existing alternatives towards combining both worlds into novel KG chatbots. Our framework compares two representative conversational models, ChatGPT and Galactica, against KGQAN, the current state-of-the-art QAS. We conduct a thorough evaluation using four real KGs across various application domains to identify the current limitations of each category of systems. Based on our findings, we propose open research opportunities to empower QASs with chatbot capabilities for KGs. All benchmarks and all raw results are available1 for further analysis.

Motivation & Objective

• Assess the strengths and limitations of conversational AI models (e.g., ChatGPT, Galactica) versus traditional KG question answering systems (KGQAn, EDGQA) for knowledge graphs.
• Evaluate performance across four real KGs from general and academic domains using a unified framework.
• Identify capabilities needed to realize KG chatbots that blend conversational AI with KGQA.
• Propose open research challenges to advance dialogue-enabled, up-to-date KG QA systems.

Proposed method

• Define a unified comparative framework with criteria including correctness, robustness, determinism, explainability, question understanding, incorporation of recent information, and domain generality.
• Select two representative conversational models (ChatGPT and Galactica) and two KGQA systems (KGQAn and EDGQA) for evaluation.
• Evaluate on four real KGs (QALD-9, YAGO, DBLP, MAG) using 450 English questions spanning boolean, listing, WH-type, and more complex queries.
• Use micro-F1 and related metrics to assess correctness; perform manual evaluation for language models due to output format differences.
• Test multiple ChatGPT variants (Default, Follow up, Excel) to study impact on recall and list completion.
• Compare determinism by repeating questions; assess robustness via spelling/grammar perturbations; analyze explainability and question understanding.

Experimental results

Research questions

• RQ1 How do ChatGPT and Galactica compare to KGQA systems in answering KG-based questions across general and academic domains?
• RQ2 What are the strengths and weaknesses of conversational AI versus traditional KGQA in terms of correctness, recall, precision, and explainability?
• RQ3 Can a combination of language models and KGQA techniques yield better generality across domains and recency of information?
• RQ4 What open challenges exist for developing KG chatbots that support dialogue, robustness, explainability, and up-to-date information?

Key findings

• KGQAn achieves high precision and recall across general and academic KGs, showing strong domain generality.
• ChatGPT attains higher precision than KGQAn on QALD-9 and YAGO but struggles with recall, especially for long lists.
• ChatGPT variants (Excel, Follow up) improve recall and list completion, sometimes surpassing KGQAn on certain benchmarks.
• Galactica underperforms on list questions, with low recall across benchmarks.
• ChatGPT demonstrates high explainability and robustness, whereas traditional KGQA systems offer deterministic, up-to-date retrieval but lack explanations.
• KGQAn maintains strong performance across KGs and supports rapid incorporation of recent information from a given KG.