[Paper Review] Make a Choice! Knowledge Base Question Answering with In-Context Learning

Chuanyuan Tan, Yuehe Chen+2 more|arXiv (Cornell University)|May 23, 2023

Topic Modeling4 citations

TL;DR

This paper proposes McL-KBQA, a knowledge base question answering framework that leverages in-context learning (ICL) with large language models (LLMs) to improve generalization in low-resource settings. By transforming questions into multiple-choice formats using a rank-based method and applying ICL with chain-of-thought prompting, the framework combines LLM predictions with existing KBQA methods via a fusion strategy, achieving state-of-the-art performance, especially under few-shot settings on WebQSP and GrailQA.

ABSTRACT

Question answering over knowledge bases (KBQA) aims to answer factoid questions with a given knowledge base (KB). Due to the large scale of KB, annotated data is impossible to cover all fact schemas in KB, which poses a challenge to the generalization ability of methods that require a sufficient amount of annotated data. Recently, LLMs have shown strong few-shot performance in many NLP tasks. We expect LLM can help existing methods improve their generalization ability, especially in low-resource situations. In this paper, we present McL-KBQA, a framework that incorporates the few-shot ability of LLM into the KBQA method via ICL-based multiple choice and then improves the effectiveness of the QA tasks. Experimental results on two KBQA datasets demonstrate the competitive performance of McL-KBQA with strong improvements in generalization. We expect to explore a new way to QA tasks from KBQA in conjunction with LLM, how to generate answers normatively and correctly with strong generalization.

Motivation & Objective

• Address the generalization challenge in KBQA due to limited annotated data covering long-tail facts in large knowledge bases.
• Leverage the few-shot reasoning ability of large language models (LLMs) to improve KBQA performance in low-resource scenarios.
• Mitigate hallucination and answer format mismatch issues common when directly prompting LLMs for KBQA by framing the task as multiple-choice selection.
• Develop a result fusion strategy that combines outputs from a rank-based KBQA method and LLM predictions to improve overall accuracy.
• Improve LLM reasoning for complex questions with constraints by incorporating chain-of-thought (CoT) explanations in the in-context learning prompt.

Proposed method

• Transform original factoid questions into multiple-choice formats using candidate logical forms generated by a rank-based KBQA method.
• Construct in-context learning (ICL) prompts by sampling few-shot examples, including question, options, correct answer, and corresponding logical forms.
• Enhance ICL with chain-of-thought (CoT) prompting to extract constraint information and answer type from questions, improving reasoning quality.
• Automatically extract the selected option letter from LLM-generated outputs using regular expressions to map to the corresponding logical form.
• Implement a result fusion strategy that selects the LLM prediction when the ranker's confidence is low, otherwise defaults to the ranker’s output.
• Execute the final logical form on the knowledge base to retrieve the answer, combining the strengths of both methods.

Experimental results

Research questions

• RQ1 Can in-context learning with LLMs improve KBQA generalization in low-resource settings where annotated data is scarce?
• RQ2 Does transforming KBQA into a multiple-choice format via a rank-based method improve LLM performance and reduce hallucination?
• RQ3 To what extent does chain-of-thought prompting enhance LLM reasoning in KBQA when used within an ICL framework?
• RQ4 How effective is the proposed result fusion strategy in combining predictions from a rank-based method and LLM to improve overall accuracy?
• RQ5 Can the framework achieve consistent performance gains across different few-shot and full-shot settings on standard KBQA benchmarks?

Key findings

• The LLM-based component outperforms the rank-based method on 19.28% of questions in the 5% few-shot setting on WebQSP, indicating strong complementary potential.
• Result fusion increases the proportion of questions where the final output outperforms both individual components by 0.61% (5% setting) and 2.01% (full setting).
• Chain-of-thought (CoT) prompting consistently improves ICL performance, with 93.52% of LLM outputs successfully matching a valid option letter in the CoT setting.
• The automatic option matching method achieves 96.36% success rate in matching valid option letters for ICL and 93.52% for CoT, demonstrating robustness.
• Despite LLMs generating hallucinated or incorrect answers in some cases, the framework reduces such failures through fusion and constraint-aware reasoning via CoT.
• The framework achieves competitive performance on both WebQSP and GrailQA, with significant improvements under few-shot conditions, validating its effectiveness in low-resource KBQA.