[Paper Review] Putting Question-Answering Systems into Practice: Transfer Learning for Efficient Domain Customization

Bernhard Kratzwald, Stefan Feuerriegel|arXiv (Cornell University)|Apr 19, 2018

Topic Modeling68 references4 citations

TL;DR

This paper proposes a transfer learning approach to efficiently customize question-answering (QA) systems for new domains using minimal labeled data. By combining metadata filtering and a novel fuse-and-oversample transfer learning method, the system achieves significant performance gains—improving answer accuracy from 1 correct answer per 3.4 questions to 1 per 2.0—using fewer than 400 annotated question-answer pairs in finance and film industry use cases.

ABSTRACT

Traditional information retrieval (such as that offered by web search engines) impedes users with information overload from extensive result pages and the need to manually locate the desired information therein. Conversely, question-answering systems change how humans interact with information systems: users can now ask specific questions and obtain a tailored answer - both conveniently in natural language. Despite obvious benefits, their use is often limited to an academic context, largely because of expensive domain customizations, which means that the performance in domain-specific applications often fails to meet expectations. This paper proposes cost-efficient remedies: (i) we leverage metadata through a filtering mechanism, which increases the precision of document retrieval, and (ii) we develop a novel fuse-and-oversample approach for transfer learning in order to improve the performance of answer extraction. Here knowledge is inductively transferred from a related, yet different, tasks to the domain-specific application, while accounting for potential differences in the sample sizes across both tasks. The resulting performance is demonstrated with actual use cases from a finance company and the film industry, where fewer than 400 question-answer pairs had to be annotated in order to yield significant performance gains. As a direct implication to management, this presents a promising path to better leveraging of knowledge stored in information systems.

Motivation & Objective

• To address the high cost and low practical adoption of domain-specific QA systems due to expensive manual customization.
• To reduce the need for large annotated datasets in domain adaptation by leveraging transfer learning.
• To improve QA performance in low-resource domains through metadata filtering and a novel transfer learning strategy.
• To demonstrate the feasibility and effectiveness of fine-tuning pre-trained QA models with minimal human-annotated data in real-world enterprise settings.

Proposed method

• Leverages metadata (e.g., document type, source) to filter and rank candidate documents before answer extraction, improving retrieval precision.
• Introduces a 'fuse-and-oversample' transfer learning method that combines knowledge from a general-domain pre-trained model with fine-tuning on a small, domain-specific dataset.
• Uses a multi-task learning setup where the model is jointly trained on general and domain-specific data, with class-imbalance mitigation via oversampling of rare domain classes.
• Employs neural network architectures (DrQA, BiDAF, R-Net) with shared embedding layers and attention mechanisms to fuse question and context representations.
• Applies word-level and character-level embeddings (using GloVe) to improve robustness to out-of-vocabulary and rare words.
• Uses bidirectional recurrent networks (LSTM/GRU) and attention mechanisms to model sequential dependencies and focus on relevant context segments.

Experimental results

Research questions

• RQ1 Can metadata filtering improve document retrieval precision in domain-specific QA systems?
• RQ2 How effective is transfer learning in reducing the data annotation burden for domain-specific QA systems?
• RQ3 Can a fuse-and-oversample strategy mitigate performance degradation when fine-tuning on small, imbalanced domain datasets?
• RQ4 To what extent can a few hundred labeled QA pairs significantly improve answer extraction accuracy in low-resource domains?

Key findings

• The proposed system improved answer extraction accuracy from 1 correct answer per 3.4 questions to 1 per 2.0 questions in real-world use cases.
• Metadata filtering significantly increased the precision of document retrieval, reducing irrelevant document retrieval and improving downstream answer quality.
• The fuse-and-oversample transfer learning method achieved substantial performance gains even with fewer than 400 annotated question-answer pairs in both finance and film domains.
• The attention mechanism that jointly models question and answer improved alignment between query and context, especially for complex or ambiguous questions.
• The use of character-level embeddings enhanced robustness to rare or out-of-vocabulary words, particularly in domain-specific terminology.
• The system demonstrated strong generalization across domains, showing that transfer learning with minimal data can yield high performance in enterprise knowledge management systems.