[Paper Review] StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

Adam Liška, Tomáš Kočiský+12 more|arXiv (Cornell University)|May 23, 2022

Topic Modeling16 citations

TL;DR

StreamingQA introduces a large-scale, time-stamped question answering benchmark using 14 years of news articles to evaluate how QA models adapt to new knowledge over time. The study demonstrates that incremental fine-tuning reduces computational costs by 95% compared to full retraining, while semi-parametric models benefit significantly from updated language models and show complementary strengths depending on named entity frequency.

ABSTRACT

Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia. However, our world is dynamic, evolves over time, and our models' knowledge becomes outdated. To study how semi-parametric QA models and their underlying parametric language models (LMs) adapt to evolving knowledge, we construct a new large-scale dataset, StreamingQA, with human written and generated questions asked on a given date, to be answered from 14 years of time-stamped news articles. We evaluate our models quarterly as they read new articles not seen in pre-training. We show that parametric models can be updated without full retraining, while avoiding catastrophic forgetting. For semi-parametric models, adding new articles into the search space allows for rapid adaptation, however, models with an outdated underlying LM under-perform those with a retrained LM. For questions about higher-frequency named entities, parametric updates are particularly beneficial. In our dynamic world, the StreamingQA dataset enables a more realistic evaluation of QA models, and our experiments highlight several promising directions for future research.

Motivation & Objective

• To address the gap in evaluating QA models' ability to adapt to new knowledge and avoid forgetting over time.
• To create a benchmark with temporal grounding for both questions and knowledge sources, unlike static snapshots like Wikipedia.
• To study the effectiveness of incremental fine-tuning and retrieval-based adaptation in parametric and semi-parametric QA models.
• To evaluate model performance on questions with relative time references and diverse named entity frequencies.

Proposed method

• Construct a large-scale dataset, StreamingQA, with human-written and LM-generated questions paired with time-stamped news articles from 2007–2020.
• Ground each question with a specific question date and provide three reference answers and evidence documents with publication dates.
• Evaluate models quarterly as they ingest new articles not seen during pre-training, simulating a streaming knowledge update scenario.
• Apply automatic filtering to remove trivial or low-quality generated questions using LM-based QA verification, Google Search validation, and named entity constraints.
• Implement incremental fine-tuning for parametric models and evaluate performance on both recent and past knowledge.
• Use frequency-based analysis to compare performance on high- vs. low-frequency named entities, revealing complementary strengths between parametric and semi-parametric approaches.

Experimental results

Research questions

• RQ1 How do parametric and semi-parametric QA models adapt to new knowledge over time without catastrophic forgetting?
• RQ2 To what extent does incremental fine-tuning of the parametric language model improve performance compared to full retraining?
• RQ3 How does the frequency of named entities affect the performance of parametric vs. semi-parametric adaptation strategies?
• RQ4 Can models effectively handle relative time references (e.g., 'a week ago') in a streaming QA setup?
• RQ5 What is the impact of using outdated language models in semi-parametric retrieval-based systems?

Key findings

• Incremental fine-tuning reduces training steps by 95% compared to full retraining while avoiding catastrophic forgetting in parametric models.
• Semi-parametric models with updated language models outperform those with outdated LMs, even when new articles are added to the retrieval corpus.
• Parametric adaptation significantly improves performance on questions involving high-frequency named entities, where retrieval-based methods struggle.
• Semi-parametric adaptation is more effective for low-frequency named entities, where retrieval noise and ambiguity are reduced.
• The benchmark reveals complementary strengths: parametric updates help with frequent entities, while retrieval-based updates help with rare ones.
• The dataset enables realistic evaluation of QA systems in dynamic, real-world settings with temporal grounding of both questions and knowledge sources.