[Paper Review] CO-Search: COVID-19 Information Retrieval with Semantic Search, Question Answering, and Abstractive Summarization

Andre Esteva, Anuprit Kale+5 more|arXiv (Cornell University)|Jun 17, 2020

Topic Modeling26 references32 citations

TL;DR

CO-Search is a retriever-ranker semantic search engine for COVID-19 literature that combines SBERT embeddings with TF-IDF and BM25, augmented by a multi-hop QA module and abstractive summarization to rank and present document answers.

ABSTRACT

The COVID-19 global pandemic has resulted in international efforts to understand, track, and mitigate the disease, yielding a significant corpus of COVID-19 and SARS-CoV-2-related publications across scientific disciplines. As of May 2020, 128,000 coronavirus-related publications have been collected through the COVID-19 Open Research Dataset Challenge. Here we present CO-Search, a retriever-ranker semantic search engine designed to handle complex queries over the COVID-19 literature, potentially aiding overburdened health workers in finding scientific answers during a time of crisis. The retriever is built from a Siamese-BERT encoder that is linearly composed with a TF-IDF vectorizer, and reciprocal-rank fused with a BM25 vectorizer. The ranker is composed of a multi-hop question-answering module, that together with a multi-paragraph abstractive summarizer adjust retriever scores. To account for the domain-specific and relatively limited dataset, we generate a bipartite graph of document paragraphs and citations, creating 1.3 million (citation title, paragraph) tuples for training the encoder. We evaluate our system on the data of the TREC-COVID information retrieval challenge. CO-Search obtains top performance on the datasets of the first and second rounds, across several key metrics: normalized discounted cumulative gain, precision, mean average precision, and binary preference.

Motivation & Objective

• Provide an effective retrieval system for the rapidly growing COVID-19 literature corpus (CORD-19).
• Integrate semantic and keyword-based retrieval signals for robust document ranking.
• Enhance ranking with multi-hop QA outputs and abstractive summaries to improve answerability.
• Train domain-aware embeddings using paragraph-citation bipartite graphs to improve semantic retrieval.
• Evaluate performance against TREC-COVID benchmarks and release open-source code.

Proposed method

• Create a bipartite graph of paragraphs and citations to generate 1.3 million (paragraph, title) training tuples for SBERT.
• Embed queries and documents with SBERT to enable semantic nearest-neighbor retrieval.
• Linearly combine SBERT paragraph scores with TF-IDF document scores and fuse with BM25 via reciprocal rank fusion.
• Use a multi-hop QA model to extract answer spans and modulate rankings based on QA outputs.
• Train an abstractive summarizer (BERT encoder + modified GPT-2 decoder) to generate a single cross-attention based summary used in ranking.

Experimental results

Research questions

• RQ1 Can a retriever-ranker model that fuses semantic, TF-IDF, and BM25 signals improve COVID-19 literature retrieval?
• RQ2 Does incorporating multi-hop QA and abstractive summaries improve the relevance and usefulness of retrieved documents?
• RQ3 How does a paragraph-citation bipartite graph-based SBERT training influence semantic retrieval in a small-to-medium domain dataset?
• RQ4 What is the impact of QA-driven and summary-driven modulation on final ranking performance?

Key findings

• CO-Search achieves top performance on several metrics (nDCG@10, P@5, P@10, MAP, Bpref) in Round 1 among automated systems.
• In Round 2, CO-Search ranks top among automated systems for the same metrics and also ranks highly when compared to all systems (including non-automated) across rounds.
• When evaluated against all topic-document pairs (annotated and non-annotated), CO-Search ranks in the top 21 in Round 1 and top 3 in Round 2.
• The system is automated and open-source, designed to support research and practical search needs during the COVID-19 crisis.
• The architecture combines semantic paragraph embeddings with keyword document representations and uses a QA-guided and summary-guided re-ranking approach.