[Paper Review] Knowledge Graph - Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Ankush Agarwal, Raj Gite+7 more|arXiv (Cornell University)|May 31, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a hybrid Knowledge Graph (KG) and deep learning (DL)-based Question Answering (QA) system for aviation safety, integrating NTSB accident reports into a domain-specific KG and combining it with BERT-QA and GPT-3-QA models. The hybrid system achieves up to 40.3% higher accuracy than BERT-QA alone and 29.3% higher than KGQA alone, demonstrating that KG and DL complement each other in improving QA performance for complex aviation safety queries.

ABSTRACT

In the commercial aviation domain, there are a large number of documents, like, accident reports (NTSB, ASRS) and regulatory directives (ADs). There is a need for a system to access these diverse repositories efficiently in order to service needs in the aviation industry, like maintenance, compliance, and safety. In this paper, we propose a Knowledge Graph (KG) guided Deep Learning (DL) based Question Answering (QA) system for aviation safety. We construct a Knowledge Graph from Aircraft Accident reports and contribute this resource to the community of researchers. The efficacy of this resource is tested and proved by the aforesaid QA system. Natural Language Queries constructed from the documents mentioned above are converted into SPARQL (the interface language of the RDF graph database) queries and answered. On the DL side, we have two different QA models: (i) BERT QA which is a pipeline of Passage Retrieval (Sentence-BERT based) and Question Answering (BERT based), and (ii) the recently released GPT-3. We evaluate our system on a set of queries created from the accident reports. Our combined QA system achieves 9.3% increase in accuracy over GPT-3 and 40.3% increase over BERT QA. Thus, we infer that KG-DL performs better than either singly.

Motivation & Objective

• To address the challenge of efficiently retrieving and understanding complex safety-related information from large, unstructured aviation documents such as NTSB accident reports and ADs.
• To construct a domain-specific Knowledge Graph from NTSB accident reports to represent entities and relationships related to aircraft accidents.
• To develop a hybrid Question Answering system that combines the strengths of KG-based SPARQL querying and deep learning-based text extraction for improved accuracy.
• To evaluate the performance of the hybrid system against standalone KG and DL QA models on a curated set of natural language questions.

Proposed method

• Constructed an Aviation Knowledge Graph from NTSB accident reports using information extraction techniques and domain-specific ontologies to represent entities (e.g., aircraft, pilot, cause) and relations.
• Implemented a KG-based QA module that translates natural language questions into SPARQL queries using a rule-based NL2SPARQL pipeline and retrieves answers from the RDF database.
• Deployed two DL-based QA models—BERT-QA and GPT-3-QA—on raw text from accident reports to extract answers directly from unstructured documents.
• Combined the outputs of the KGQA and DLQA modules by merging top-10 answers from both systems, leveraging the complementary strengths of structured and unstructured data processing.
• Evaluated the system using handcrafted questions from accident reports, measuring performance with exact match, semantic accuracy, and semantic recall metrics.
• Used a re-ranking or combination framework to improve final answer selection, with GPT-3-QA showing superior performance in complex reasoning tasks.

Experimental results

Research questions

• RQ1 Can a Knowledge Graph constructed from NTSB accident reports effectively support structured, relationship-based question answering in the aviation safety domain?
• RQ2 How does the integration of a KG with deep learning models improve QA accuracy compared to using either component in isolation?
• RQ3 To what extent do BERT-QA and GPT-3-QA models differ in their ability to answer complex, multi-hop questions in aviation safety documents?
• RQ4 What are the limitations of NL2SPARQL pipelines and how do they affect KG-based QA performance?
• RQ5 Can a hybrid system overcome the coverage and reasoning limitations of standalone KG and DL QA modules?

Key findings

• The hybrid KGQA + BERT-QA system achieved a 7% increase in accuracy over the standalone KGQA module and a 40.3% increase over the standalone BERT-QA model.
• The hybrid KGQA + GPT3-QA system achieved a 29.3% increase in accuracy over KGQA and a 9.3% increase over GPT3-QA alone, making it the best-performing configuration.
• The KGQA module was limited by the inability of the NL2SPARQL pipeline to handle complex or non-triple-formatted queries, such as those requiring multi-sentence reasoning.
• The BERT-QA model underperformed due to lack of domain-specific fine-tuning, indicating the need for domain-adapted pre-training.
• The GPT-3-QA model struggled with multi-hop questions requiring information from multiple paragraphs, highlighting a key limitation in long-context reasoning.
• The combined system outperformed individual components by leveraging the complementary strengths of structured KG reasoning and unstructured text understanding.