[Paper Review] The TechQA Dataset

Vittorio Castelli, Rishav Chakravarti+19 more|arXiv (Cornell University)|Nov 8, 2019

Topic Modeling14 references4 citations

TL;DR

TechQA is a domain-adaptation question answering dataset comprising 600 training, 310 development, and 490 evaluation question-answer pairs drawn from real user questions in IBM technical forums, paired with accepted answers from official IBM Technote documents. It enables research in domain adaptation by providing authentic, real-world technical support queries and a companion collection of 801,998 publicly available Technotes for pretraining and domain representation learning.

ABSTRACT

We introduce TechQA, a domain-adaptation question answering dataset for the technical support domain. The TechQA corpus highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum, rather than questions generated specifically for a competition or a task. Second, it has a real-world size -- 600 training, 310 dev, and 490 evaluation question/answer pairs -- thus reflecting the cost of creating large labeled datasets with actual data. Consequently, TechQA is meant to stimulate research in domain adaptation rather than being a resource to build QA systems from scratch. The dataset was obtained by crawling the IBM Developer and IBM DeveloperWorks forums for questions with accepted answers that appear in a published IBM Technote---a technical document that addresses a specific technical issue. We also release a collection of the 801,998 publicly available Technotes as of April 4, 2019 as a companion resource that might be used for pretraining, to learn representations of the IT domain language.

Motivation & Objective

• To address the lack of realistic, large-scale labeled datasets in the technical support domain for domain adaptation research.
• To provide a dataset that reflects real-world user queries and production-level technical documentation, rather than synthetic or competition-specific questions.
• To support research in adapting NLP models to the specialized language and information needs of IT support contexts.
• To release a companion corpus of 801,998 publicly available IBM Technotes for pretraining and learning domain-specific representations.

Proposed method

• Crawling IBM Developer and IBM DeveloperWorks forums to collect questions that have accepted answers linked to official IBM Technote documents.
• Selecting only questions with corresponding, publicly available IBM Technotes to ensure factual grounding and domain relevance.
• Curating the final dataset into 600 training, 310 development, and 490 evaluation splits to reflect real-world data collection costs and scale.
• Releasing the 801,998 available Technotes as a companion resource for pretraining and learning IT domain language representations.
• Using the dataset to evaluate domain adaptation techniques rather than building QA systems from scratch.
• Focusing on real user questions and production documentation to ensure authenticity and practical relevance.

Experimental results

Research questions

• RQ1 How effective are domain adaptation techniques when applied to real user questions from technical support forums?
• RQ2 To what extent can pretraining on the companion Technote corpus improve performance on the TechQA dataset?
• RQ3 How does the performance of models fine-tuned on TechQA compare to those trained on general-domain QA datasets in the technical support domain?
• RQ4 What impact does the use of real-world, production-level documentation (Technotes) have on model generalization in technical QA?
• RQ5 Can models trained on TechQA generalize to unseen technical support queries beyond the provided splits?

Key findings

• The TechQA dataset contains 600 training, 310 development, and 490 evaluation question-answer pairs, reflecting realistic data collection costs and scale.
• The dataset is built from actual user questions in IBM forums that are linked to official IBM Technote documents, ensuring factual accuracy and real-world relevance.
• A companion corpus of 801,998 publicly available IBM Technotes was released to support pretraining and domain representation learning.
• The dataset is explicitly designed to stimulate research in domain adaptation rather than serve as a standalone resource for training QA systems from scratch.
• The inclusion of real technical documentation (Technotes) enables better modeling of IT domain language and technical reasoning.
• The dataset supports evaluation of models in a realistic technical support setting, emphasizing the importance of domain-specific knowledge and natural language understanding.