[Paper Review] ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collections

Jiexin Wang, Adam Jatowt+1 more|arXiv (Cornell University)|Sep 8, 2021

Topic Modeling4 citations

TL;DR

ArchivalQA introduces a large-scale open-domain question answering (ODQA) benchmark with 532,444 question-answer pairs over the New York Times Annotated Corpus (1987–2007), using a semi-automatic pipeline combining neural question generation and multi-stage filtering to create high-quality, temporally grounded questions. The dataset is split into sub-datasets by difficulty and temporal expression presence, enabling fine-grained evaluation of ODQA models on historical news, with DPR models showing strong performance on hard, low-lexical-overlap questions.

ABSTRACT

In the last few years, open-domain question answering (ODQA) has advanced rapidly due to the development of deep learning techniques and the availability of large-scale QA datasets. However, the current datasets are essentially designed for synchronic document collections (e.g., Wikipedia). Temporal news collections such as long-term news archives spanning several decades, are rarely used in training the models despite they are quite valuable for our society. To foster the research in the field of ODQA on such historical collections, we present ArchivalQA, a large question answering dataset consisting of 532,444 question-answer pairs which is designed for temporal news QA. We divide our dataset into four subparts based on the question difficulty levels and the containment of temporal expressions, which we believe are useful for training and testing ODQA systems characterized by different strengths and abilities. The novel QA dataset-constructing framework that we introduce can be also applied to generate non-ambiguous questions of good quality over other types of temporal document collections.

Motivation & Objective

• To address the lack of large-scale, high-quality ODQA datasets for long-term historical news archives, which are underutilized despite their societal value.
• To develop a scalable, semi-automatic framework for generating diverse, non-ambiguous questions over temporal document collections using neural question generation and filtering.
• To create a benchmark that captures the challenges of temporal reasoning, including temporal ambiguity and lexical variation across decades.
• To enable fine-grained evaluation of ODQA systems by differentiating questions based on difficulty and presence of temporal expressions.
• To support research in temporal information retrieval, historical fact-finding, and education through a rich, diverse, and temporally grounded dataset.

Proposed method

• Leverage neural question generation models to automatically produce a large pool of candidate questions from news articles in the NYT corpus (1987–2007).
• Apply a cascade of filtering steps—based on relevance, ambiguity, and factual consistency—to remove low-quality questions and retain high-quality QA instances.
• Structure the final dataset into four sub-datasets: ArchivalQAEasy, ArchivalQAHard, ArchivalQATime, and ArchivalQANoTime, based on question difficulty and temporal expression presence.
• Use the NYT corpus as the underlying document collection, ensuring temporal consistency and historical depth for training and evaluation.
• Split each sub-dataset into training (80%), development (10%), and test (10%) sets for standardized evaluation.
• Evaluate multiple ODQA models, including sparse retrievers (e.g., BERTserini) and dense retrievers (e.g., DPR), on the sub-datasets to analyze performance differences.

Experimental results

Research questions

• RQ1 How effective are existing ODQA models in answering detailed, historically specific questions over long-term news archives?
• RQ2 To what extent does the presence of temporal expressions in questions affect model performance on historical QA tasks?
• RQ3 Can dense retrieval models (e.g., DPR) outperform sparse retrieval models on hard, low-lexical-overlap questions in temporal document collections?
• RQ4 How does question difficulty, as defined by complexity and lexical overlap, influence ODQA system performance?
• RQ5 Can a semi-automatic pipeline with multi-stage filtering generate high-quality, non-ambiguous QA pairs suitable for training and evaluating ODQA systems on archival data?

Key findings

• DPR models outperform sparse retrievers on ArchivalQAHard, achieving a 54.85% improvement in exact match (EM) and 40.35% in F1, indicating strong generalization on low-lexical-overlap questions.
• BERTserini-NYT-TempRes shows a 106.83% improvement in EM on ArchivalQAHard compared to ArchivalQAEasy, confirming the increased difficulty of hard questions.
• Models perform slightly better on ArchivalQANoTime than on ArchivalQATime, suggesting that temporal signals may currently be underutilized or even detrimental when not properly modeled.
• The semi-automatic framework successfully generated 532,444 high-quality QA pairs, with sub-datasets enabling fine-grained evaluation based on difficulty and temporal content.
• The dataset reveals that Wikipedia-based knowledge is insufficient for answering detailed, minor historical events, underscoring the need for archival news sources in ODQA.
• The performance gap between easy and hard questions highlights the importance of temporal reasoning and context understanding in historical QA systems.