[Paper Review] A Hybrid Citation Retrieval Algorithm for Evidence-based Clinical Knowledge Summarization: Combining Concept Extraction, Vector Similarity and Query Expansion for High Precision

Kalpana Raja, Andrew J. Sauer+3 more|arXiv (Cornell University)|Sep 6, 2016

Topic Modeling1 references3 citations

TL;DR

This paper proposes a hybrid citation retrieval system that integrates concept extraction, vector similarity, and query expansion to improve precision in retrieving relevant medical citations for evidence-based clinical knowledge summarization. The system achieves an F-score of 41.2% on heart failure and 42.4% on atrial fibrillation topics—significantly outperforming baselines—while retrieving up to 18 new relevant citations per topic.

ABSTRACT

Novel information retrieval methods to identify citations relevant to a clinical topic can overcome the knowledge gap existing between the primary literature (MEDLINE) and online clinical knowledge resources such as UpToDate. Searching the MEDLINE database directly or with query expansion methods returns a large number of citations that are not relevant to the query. The current study presents a citation retrieval system that retrieves citations for evidence-based clinical knowledge summarization. This approach combines query expansion, concept-based screening algorithm, and concept-based vector similarity. We also propose an information extraction framework for automated concept (Population, Intervention, Comparison, and Disease) extraction. We evaluated our proposed system on all topics (as queries) available from UpToDate for two diseases, heart failure (HF) and atrial fibrillation (AFib). The system achieved an overall F-score of 41.2% on HF topics and 42.4% on AFib topics on a gold standard of citations available in UpToDate. This is significantly high when compared to a query-expansion based baseline (F-score of 1.3% on HF and 2.2% on AFib) and a system that uses query expansion with disease hyponyms and journal names, concept-based screening, and term-based vector similarity system (F-score of 37.5% on HF and 39.5% on AFib). Evaluating the system with top K relevant citations, where K is the number of citations in the gold standard achieved a much higher overall F-score of 69.9% on HF topics and 75.1% on AFib topics. In addition, the system retrieved up to 18 new relevant citations per topic when tested on ten HF and six AFib clinical topics.

Motivation & Objective

• Address the knowledge gap between primary medical literature (MEDLINE) and clinical knowledge resources like UpToDate.
• Reduce the high volume of irrelevant citations returned by direct MEDLINE searches or basic query expansion.
• Improve precision in citation retrieval for evidence-based clinical knowledge summarization.
• Automatically extract key clinical concepts (Population, Intervention, Comparison, Disease) from clinical queries.
• Develop a scalable, hybrid retrieval framework that enhances recall and precision through multi-stage filtering and ranking.

Proposed method

• Extract clinical concepts (PICO: Population, Intervention, Comparison, Outcome) from clinical queries using an information extraction framework.
• Apply query expansion using synonyms and related terms to enhance recall without sacrificing precision.
• Use concept-based vector similarity to compute semantic relevance between query concepts and citation abstracts.
• Combine concept-based screening with term-based vector similarity in a hybrid ranking model.
• Rank citations using a weighted fusion of concept-based and vector-based similarity scores.
• Evaluate system performance using gold-standard citation sets from UpToDate for heart failure and atrial fibrillation topics.

Experimental results

Research questions

• RQ1 Can a hybrid retrieval system combining concept extraction, query expansion, and vector similarity significantly improve citation retrieval precision for clinical topics?
• RQ2 How does the integration of PICO concept extraction enhance retrieval performance compared to traditional keyword-based methods?
• RQ3 To what extent does query expansion improve recall while maintaining high precision in clinical citation retrieval?
• RQ4 How does the system perform when retrieving top-K citations relative to gold-standard references?
• RQ5 Can the system identify novel, relevant citations not present in existing clinical knowledge resources?

Key findings

• The system achieved an F-score of 41.2% on heart failure topics and 42.4% on atrial fibrillation topics, significantly outperforming a query-expansion-only baseline (1.3% and 2.2% F-score, respectively).
• When retrieving the top-K citations (matching gold-standard counts), the system achieved an F-score of 69.9% for heart failure and 75.1% for atrial fibrillation, indicating strong performance on high-priority results.
• The system retrieved up to 18 new relevant citations per topic that were not present in the UpToDate gold standard, demonstrating its ability to discover novel evidence.
• The hybrid approach—combining concept-based screening with vector similarity—yielded an F-score of 37.5% on HF and 39.5% on AFib, showing incremental improvement over baseline methods.
• The integration of concept extraction and vector similarity significantly enhanced precision, reducing irrelevant results compared to keyword-only or expansion-only strategies.
• The system’s performance was robust across diverse clinical topics, with consistent gains in F-score and top-K retrieval accuracy for both heart failure and atrial fibrillation.