[Paper Review] KITLM: Domain-Specific Knowledge InTegration into Language Models for Question Answering

Ankush Agarwal, Sakharam Gawade+2 more|arXiv (Cornell University)|Aug 7, 2023

Topic Modeling4 citations

TL;DR

KITLM proposes a knowledge infusion framework that enhances domain-specific question answering by integrating relevant knowledge graph triples into T5-based language models through a novel fine-tuning objective. By filtering and injecting only pertinent structured knowledge during training and inference, KITLM achieves over 1.5× higher exact match scores than GPT-3.5-turbo and SOTA methods on MetaQA and AeroQA, while reducing model size and computational cost.

ABSTRACT

Large language models (LLMs) have demonstrated remarkable performance in a wide range of natural language tasks. However, as these models continue to grow in size, they face significant challenges in terms of computational costs. Additionally, LLMs often lack efficient domain-specific understanding, which is particularly crucial in specialized fields such as aviation and healthcare. To boost the domain-specific understanding, we propose, KITLM, a novel knowledge base integration approach into language model through relevant information infusion. By integrating pertinent knowledge, not only the performance of the language model is greatly enhanced, but the model size requirement is also significantly reduced while achieving comparable performance. Our proposed knowledge-infused model surpasses the performance of both GPT-3.5-turbo and the state-of-the-art knowledge infusion method, SKILL, achieving over 1.5 times improvement in exact match scores on the MetaQA. KITLM showed a similar performance boost in the aviation domain with AeroQA. The drastic performance improvement of KITLM over the existing methods can be attributed to the infusion of relevant knowledge while mitigating noise. In addition, we release two curated datasets to accelerate knowledge infusion research in specialized fields: a) AeroQA, a new benchmark dataset designed for multi-hop question-answering within the aviation domain, and b) Aviation Corpus, a dataset constructed from unstructured text extracted from the National Transportation Safety Board reports. Our research contributes to advancing the field of domain-specific language understanding and showcases the potential of knowledge infusion techniques in improving the performance of language models on question-answering.

Motivation & Objective

• Address the limited domain-specific understanding of large language models (LLMs) in specialized fields like aviation and healthcare.
• Overcome the high computational cost and poor generalization of continual pre-training methods for knowledge infusion.
• Develop a lightweight, efficient method to enhance language models with structured knowledge without architectural modifications.
• Create curated datasets to support research in domain-specific knowledge infusion and multi-hop question answering.
• Demonstrate that smaller, knowledge-infused models can surpass large LLMs in domain-specific QA tasks.

Proposed method

• Propose a knowledge integration objective that verbalizes knowledge graph (KG) triples and injects them during both training and inference.
• Use ColBERTv2 to retrieve and filter the most relevant KG triples for each question-answer pair, minimizing noise.
• Fine-tune T5-based models using a combination of natural language question-answer pairs and injected verbalized KG triples.
• Eliminate the need for continual pre-training to avoid catastrophic forgetting and reduce computational overhead.
• Select N relevant triples for N-hop question-answering tasks, enabling scalable and efficient knowledge infusion.
• Integrate knowledge at inference time by dynamically retrieving and injecting relevant triples, ensuring contextually accurate responses.

Figure 1. The proposed framework, KITLM, is illustrated in the flow diagram. Initially, triples are extracted from knowledge bases like the WikiMovies dataset (Miller et al . , 2016 ) and transformed into verbalized form. Subsequently, ColBERTv2 (Santhanam et al . , 2022 ) is employed to retrieve th

Experimental results

Research questions

• RQ1 Can knowledge graph infusion improve the performance of smaller language models in domain-specific question answering compared to large LLMs?
• RQ2 How does the proposed knowledge infusion method compare to existing approaches like SKILL and GPT-3.5-turbo in multi-hop QA tasks?
• RQ3 To what extent does filtering relevant KG triples reduce noise and improve model generalization?
• RQ4 Can a lightweight, fine-tuning-based method outperform continual pre-training methods in terms of efficiency and performance?
• RQ5 How effective is the knowledge infusion approach in specialized domains like aviation, where domain-specific knowledge is critical?

Key findings

• KITLM achieves over 1.5× higher exact match scores than GPT-3.5-turbo and the SOTA method SKILL on the MetaQA benchmark.
• KITLM outperforms GPT-3 and ChatGPT on both 1-hop and multi-hop questions in the AeroQA dataset, with GPT-3 achieving only 18.73% EM on 1-hop MetaQA.
• The model maintains high performance across 2-hop and 3-hop QA tasks, demonstrating strong reasoning capabilities through knowledge infusion.
• KITLM reduces computational cost by eliminating continual pre-training and instead using efficient triple filtering and injection at inference.
• Fine-tuning with injected knowledge enables smaller models to surpass large LLMs in domain-specific QA, highlighting the effectiveness of knowledge infusion.
• The released AeroQA dataset and Aviation Corpus provide valuable resources for advancing research in aviation-specific multi-hop question answering and knowledge infusion.