[Paper Review] Denmark's Participation in the Search Engine TREC COVID-19 Challenge: Lessons Learned about Searching for Precise Biomedical Scientific Information on COVID-19

Lucas Chaves Lima, Casper Worm Hansen+6 more|arXiv (Cornell University)|Nov 25, 2020

Topic Modeling34 references4 citations

TL;DR

This paper details Denmark's participation in the 2020 TREC-COVID Challenge, evaluating retrieval strategies for precise biomedical information on COVID-19 using the CORD-19 dataset. It finds that standard BM25 retrieval outperformed advanced BERT-based embeddings, and that document source (e.g., PMC, Elsevier) was a strong predictor of relevance, with limited gains from deep learning due to insufficient training data.

ABSTRACT

This report describes the participation of two Danish universities, University of Copenhagen and Aalborg University, in the international search engine competition on COVID-19 (the 2020 TREC-COVID Challenge) organised by the U.S. National Institute of Standards and Technology (NIST) and its Text Retrieval Conference (TREC) division. The aim of the competition was to find the best search engine strategy for retrieving precise biomedical scientific information on COVID-19 from the largest, at that point in time, dataset of curated scientific literature on COVID-19 -- the COVID-19 Open Research Dataset (CORD-19). CORD-19 was the result of a call to action to the tech community by the U.S. White House in March 2020, and was shortly thereafter posted on Kaggle as an AI competition by the Allen Institute for AI, the Chan Zuckerberg Initiative, Georgetown University's Center for Security and Emerging Technology, Microsoft, and the National Library of Medicine at the US National Institutes of Health. CORD-19 contained over 200,000 scholarly articles (of which more than 100,000 were with full text) about COVID-19, SARS-CoV-2, and related coronaviruses, gathered from curated biomedical sources. The TREC-COVID challenge asked for the best way to (a) retrieve accurate and precise scientific information, in response to some queries formulated by biomedical experts, and (b) rank this information decreasingly by its relevance to the query. In this document, we describe the TREC-COVID competition setup, our participation to it, and our resulting reflections and lessons learned about the state-of-art technology when faced with the acute task of retrieving precise scientific information from a rapidly growing corpus of literature, in response to highly specialised queries, in the middle of a pandemic.

Motivation & Objective

• To evaluate and compare retrieval strategies for precise biomedical information on COVID-19 during a global pandemic.
• To assess the effectiveness of standard retrieval methods, state-of-the-art semantic embeddings, and meta-search heuristics in an unsupervised, evolving information retrieval setting.
• To understand the impact of data source, embedding models, and model complexity on retrieval performance with limited labeled data.
• To identify practical limitations and future research directions for biomedical IR in crisis scenarios.

Proposed method

• Employed Best Match 25 (BM25) with default parameters (k₁=1.2, b=0.75) on the CORD-19 dataset using the Indri search engine.
• Integrated BioBERT and SciBERT embeddings for semantic representation, but did not fine-tune them due to data scarcity.
• Applied meta-search heuristics combining results from multiple retrieval strategies to improve ranking precision.
• Used NDCG@K, Precision@K, Brevity Penalty, RBP (p=0.5), and MAP for evaluation with varying K, using trec_eval.
• Leveraged expert assessments from NIST to evaluate runs, with assessments released progressively across five competition rounds.
• Analyzed relevance distribution across data sources to identify strong predictors of document relevance.

Experimental results

Research questions

• RQ1 Can standard retrieval methods like BM25 achieve competitive performance on biomedical IR tasks in a crisis setting with limited labeled data?
• RQ2 To what extent do domain-specific BERT embeddings (e.g., BioBERT) improve retrieval performance over traditional term frequency-based methods in this context?
• RQ3 How significant is the contribution of document source (e.g., PMC, medRxiv, Elsevier) as a relevance signal in biomedical information retrieval?
• RQ4 What are the practical limitations of applying advanced machine learning models to biomedical IR when training data is scarce?
• RQ5 Can weak supervision from automatically generated query-document pairs improve retrieval performance in low-data regimes?

Key findings

• Standard BM25 retrieval with default parameters achieved reliable and satisfactory performance, outperforming more complex models in this setting.
• Semantic embeddings from BioBERT and SciBERT did not yield significant performance improvements over traditional word count-based representations, despite higher computational cost.
• The source of the scientific literature was a strong predictor of relevance: Elsevier, medRxiv, and PMC contributed the majority of relevant and partially relevant documents.
• Document source accounted for 73.9% of all relevant and partially relevant documents, with Elsevier alone contributing 35.48% of relevant documents.
• The dataset size (up to 200,000 documents) and limited number of queries (max 50) were insufficient to train advanced machine learning models effectively, limiting generalization.
• Fine-tuning of BERT models was not performed, which likely hindered performance; teams that did fine-tune on MS-MARCO or similar datasets achieved better results.