[Paper Review] MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval

Qiao Jin, Won Bae Kim+5 more|arXiv (Cornell University)|Jul 2, 2023

Topic ModelingComputer Science64 references9 citations

TL;DR

MedCPT introduces a contrastively pre-trained transformer model trained on 255 million PubMed click logs to enable zero-shot semantic biomedical information retrieval, achieving state-of-the-art results across six tasks.

ABSTRACT

Information retrieval (IR) is essential in biomedical knowledge acquisition and clinical decision support. While recent progress has shown that language model encoders perform better semantic retrieval, training such models requires abundant query-article annotations that are difficult to obtain in biomedicine. As a result, most biomedical IR systems only conduct lexical matching. In response, we introduce MedCPT, a first-of-its-kind Contrastively Pre-trained Transformer model for zero-shot semantic IR in biomedicine. To train MedCPT, we collected an unprecedented scale of 255 million user click logs from PubMed. With such data, we use contrastive learning to train a pair of closely-integrated retriever and re-ranker. Experimental results show that MedCPT sets new state-of-the-art performance on six biomedical IR tasks, outperforming various baselines including much larger models such as GPT-3-sized cpt-text-XL. In addition, MedCPT also generates better biomedical article and sentence representations for semantic evaluations. As such, MedCPT can be readily applied to various real-world biomedical IR tasks.

Motivation & Objective

• Motivate zero-shot semantic biomedical IR without abundant query-article annotations.
• Leverage large-scale PubMed user click logs to train a contrastive retrieval framework.
• Develop a tightly integrated retriever and re-ranker for improved semantic search in biomedicine.
• Demonstrate superior performance over baselines and much larger models on multiple biomedical IR tasks.

Proposed method

• Train a contrastive, dual-component model consisting of a retriever and a re-ranker.
• Use 255 million PubMed user click logs to supervise contrastive learning.
• Optimize with contrastive losses to align query and article representations.
• Evaluate representations on article and sentence-level semantic tasks.
• Compare against baselines including large models such as GPT-3-sized CPT-text-XL.

Experimental results

Research questions

• RQ1 Can zero-shot biomedical IR be achieved via contrastive pre-training on large-scale PubMed search logs?
• RQ2 Does a retriever–re-ranker architecture trained with click data outperform lexical baselines and larger models on biomedical IR tasks?
• RQ3 Do the learned representations improve semantic retrieval at both article and sentence levels in biomedicine?

Key findings

• MedCPT sets new state-of-the-art performance on six biomedical IR tasks.
• MedCPT outperforms various baselines including GPT-3-sized CPT-text-XL.
• MedCPT yields better biomedical article and sentence representations for semantic evaluations.