[Paper Review] Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned

Edwin Zhang, Nïkhil Gupta+3 more|arXiv (Cornell University)|Apr 10, 2020

Explainable Artificial Intelligence (XAI)Computer Science26 references35 citations

TL;DR

The authors rapidly build and deploy the Neural Covidex, a neural-ranking based search engine for the COVID-19 Open Research Dataset (CORD-19), and share modular components and lessons from a live system deployment.

ABSTRACT

We present the Neural Covidex, a search engine that exploits the latest neural ranking architectures to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI. This web application exists as part of a suite of tools that we have developed over the past few weeks to help domain experts tackle the ongoing global pandemic. We hope that improved information access capabilities to the scientific literature can inform evidence-based decision making and insight generation. This paper describes our initial efforts and offers a few thoughts about lessons we have learned along the way.

Motivation & Objective

• Motivate and describe how improved information access can aid evidence-based decision making during the COVID-19 pandemic.
• Describe the architecture, components, and rapid deployment of end-to-end search applications over CORD-19.
• Provide lessons learned to inform future crisis-driven information systems and open-source collaboration.
• Offer reusable software artifacts and notebooks to enable replication and extension by the research community.

Proposed method

• Adopt a multi-stage search architecture with an initial keyword retrieval stage followed by neural reranking.
• Index CORD-19 using a paragraph-level scheme with BM25 scoring and a paragraph-index to balance relevance and article-level deduplication.
• Provide Python bindings (Pyserini) to Anserini for fast, reproducible keyword search and integrate with a Solr/Blacklight frontend for faceted browsing.
• Fine-tune a T5-base model on MS MARCO for a binary relevance task to rerank candidate documents.
• Segment long documents into spans with a sliding window and select the highest-span relevance for final article ranking.
• Highlight salient sentences unsupervised using BioBERT to guide readers to pertinent passages.
• Deploy the Neural Covidex as a monolithic service (with plans to modularize) using FastAPI, React frontend, and a GPU-enabled server cluster with Cloudflare load balancing.

Experimental results

Research questions

• RQ1 How can neural ranking be integrated with traditional keyword search to improve information access for CORD-19?
• RQ2 What are practical considerations and trade-offs when rapidly deploying a live neural search system for a crisis-driven literature corpus?
• RQ3 What is the impact of paragraph-level versus full-text indexing on retrieval quality for CORD-19?
• RQ4 Can a pretrained sequence-to-sequence model trained on MS MARCO effectively rerank CORD-19 results?
• RQ5 What are the operational and usability lessons learned from deploying a live neural search system in a pandemic context?

Key findings

• The Neural Covidex combines BM25 keyword retrieval with a T5-base neural reranker to produce ranked results over CORD-19.
• A paragraph-level index improves retrieval balance and duplicates across paragraphs can provide signals for downstream components.
• End-to-end latency for a typical query is around two seconds on a small GPU cluster, enabling interactive use.
• Open-source artifacts (Anserini, Pyserini, and notebooks) were released to support replication and extension by the research community.
• The authors emphasize that formal end-to-end evaluations are premature without user-centered feedback and that informal usability discussions are valuable for crisis-time tools.
• Lessons highlight the importance of open-source ecosystems, software engineering practices, and deployment considerations (latency, throughput, UI polish) for live systems.