[论文解读] KQA Pro: A Large-Scale Dataset with Interpretable Programs and Accurate SPARQLs for Complex Question Answering over Knowledge Base

Jiaxin Shi, Shulin Cao+6 more|arXiv (Cornell University)|Jul 8, 2020

Topic Modeling参考文献 53被引用 5

一句话总结

KQA Pro 引入了一个大规模、多样化的知识库问答（KBQA）数据集，包含120K个复杂问题，附带可解释的推理程序和准确的SPARQL查询。该数据集使最先进的模型在使用自然语言问题到SPARQL或自然语言问题到程序监督进行训练时，KBQA准确率可达到约90%，显著优于仅使用问答的基线模型，并接近人类专家水平。

ABSTRACT

Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation, and etc. Existing benchmarks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are either generated by templates, leading to poor diversity, or on a small scale. To this end, we introduce KQA Pro, a large-scale dataset for Complex KBQA. We define a compositional and highly-interpretable formal format, named Program, to represent the reasoning process of complex questions. We propose compositional strategies to generate questions, corresponding SPARQLs, and Programs with a small number of templates, and then paraphrase the generated questions to natural language questions (NLQ) by crowdsourcing, giving rise to around 120K diverse instances. SPARQL and Program depict two complementary solutions to answer complex questions, which can benefit a large spectrum of QA methods. Besides the QA task, KQA Pro can also serves for the semantic parsing task. As far as we know, it is currently the largest corpus of NLQ-to-SPARQL and NLQ-to-Program. We conduct extensive experiments to evaluate whether machines can learn to answer our complex questions in different cases, that is, with only QA supervision or with intermediate SPARQL/Program supervision. We find that state-of-the-art KBQA methods learnt from only QA pairs perform very poor on our dataset, implying our questions are more challenging than previous datasets. However, pretrained models learnt from our NLQ-to-SPARQL and NLQ-to-Program annotations surprisingly achieve about 90\% answering accuracy, which is even close to the human expert performance...

研究动机与目标

• 解决复杂知识库问答（KBQA）领域缺乏大规模、多样化且可解释数据集的问题。
• 克服现有基准缺乏推理轨迹或使用模板生成、多样性低的问题。
• 提供SPARQL和正式程序标注，以支持KBQA中的多种学习范式。
• 支持在仅问答（QA-only）和中间监督（SPARQL/程序）设置下对模型进行评估。
• 构建一个挑战当前模型并支持语义解析与组合推理研究的基准。

提出的方法

• 定义一种组合式、可解释的正式语言‘Program’，用于表示复杂问题的逐步推理过程。
• 使用少量模板结合组合策略，生成多样化的问题、SPARQL和程序。
• 通过众包将生成的问题改写为自然语言，以确保语言多样性与流畅性。
• 确保SPARQL和程序标注准确且互补，以同时支持问答与语义解析任务。
• 利用NLQ-to-SPARQL和NLQ-to-Program标注进行预训练，以提升模型泛化能力和推理准确率。
• 设计数据集以支持在仅问答和中间监督两种范式下的评估。

实验结果

研究问题

• RQ1 仅在问答对上进行训练的模型，当暴露于像KQA Pro这样大规模、多样化且具有挑战性的数据集时，能否在复杂KBQA任务中实现良好泛化？
• RQ2 与仅使用问答监督相比，通过SPARQL或程序标注进行中间监督在多大程度上提升了KBQA性能？
• RQ3 可解释的推理程序的引入在多大程度上增强了模型在复杂推理任务中的可解释性与性能？
• RQ4 在问题多样性、推理复杂性和模型性能方面，KQA Pro与现有基准相比如何？
• RQ5 当在NLQ-to-SPARQL和NLQ-to-Program标注上进行微调时，预训练模型能否在复杂KBQA上实现接近人类专家的性能？

主要发现

• 仅在问答对上进行训练的最先进KBQA模型在KQA Pro上的表现较差，表明该数据集相比以往基准具有更高的复杂性。
• 在NLQ-to-SPARQL标注上进行微调的预训练模型，其问答准确率约为90%，接近人类专家水平。
• 在NLQ-to-Program监督下训练的模型也达到了高准确率，证明了可解释推理监督的价值。
• SPARQL与程序标注的结合，相比仅使用问答监督，能够实现更鲁棒且更具泛化能力的学习。
• 该数据集的多样性和组合式结构有效挑战了当前模型，凸显了对更优推理机制的需求。
• KQA Pro是目前已知最大规模的NLQ-to-SPARQL和NLQ-to-Program映射语料库，为复杂KBQA研究设立了新基准。