[Paper Review] Graph-Based Retriever Captures the Long Tail of Biomedical Knowledge

Julien Delile, S. Mukherjee+2 more|arXiv (Cornell University)|Feb 19, 2024

Biomedical Text Mining and Ontologies5 citations

TL;DR

The paper presents a knowledge-graph–based retrieval (KG IR) approach for Retrieval-Augmented Generation in biomedicine, showing it better captures long-tail knowledge than embedding-based retrieval and combines well with embeddings in a hybrid model.

ABSTRACT

Large language models (LLMs) are transforming the way information is retrieved with vast amounts of knowledge being summarized and presented via natural language conversations. Yet, LLMs are prone to highlight the most frequently seen pieces of information from the training set and to neglect the rare ones. In the field of biomedical research, latest discoveries are key to academic and industrial actors and are obscured by the abundance of an ever-increasing literature corpus (the information overload problem). Surfacing new associations between biomedical entities, e.g., drugs, genes, diseases, with LLMs becomes a challenge of capturing the long-tail knowledge of the biomedical scientific production. To overcome this challenge, Retrieval Augmented Generation (RAG) has been proposed to alleviate some of the shortcomings of LLMs by augmenting the prompts with context retrieved from external datasets. RAG methods typically select the context via maximum similarity search over text embeddings. In this study, we show that RAG methods leave out a significant proportion of relevant information due to clusters of over-represented concepts in the biomedical literature. We introduce a novel information-retrieval method that leverages a knowledge graph to downsample these clusters and mitigate the information overload problem. Its retrieval performance is about twice better than embedding similarity alternatives on both precision and recall. Finally, we demonstrate that both embedding similarity and knowledge graph retrieval methods can be advantageously combined into a hybrid model that outperforms both, enabling potential improvements to biomedical question-answering models.

Motivation & Objective

• Motivate the information overload problem in biomedical literature and the need to surface long-tail knowledge for biomedical QA/QS.
• Introduce a graph-based retrieval method that rebalances retrieved chunks using a biomedical knowledge graph.
• Demonstrate that KG IR outperforms embedding similarity IR and that a hybrid KG+embedding approach yields superior results.
• Show how recency and impact can prioritize text chunks along KG paths to surface recent, influential discoveries.

Proposed method

• Develop two retrieval strategies: embedding similarity IR (ES IR) using sentence embeddings and cosine similarity on PubMed sentences (text-embedding-ada-002).
• Construct a biomedical knowledge graph (KG) by NER/RE processing of PubMed abstracts (KAZU + TinyBERN2 for NER; PubMedBERT fine-tuned on BioRED for RE).
• Map text chunks to KG nodes/edges by entity annotations and relations; use shortest paths between question entities to select relevant chunks.
• Prioritize retrieved chunks using a Pareto-front ranking over recency (publication year) and impact (citation count).
• Evaluate a hybrid IR (Hybrid IR) by averaging rescaled scores from ES IR and KG IR to combine strengths.
• Compare retrieval performance on eight diseases using precision@K and recall@K against a gold-standard expert-annotated set.

Experimental results

Research questions

• RQ1 DoesKG-based IR retrieve a distinct long-tail of biomedical knowledge that embedding-based IR misses?
• RQ2 How does KG IR compare to embedding IR in precision/recall for targeted biomedical questions?
• RQ3 Can a hybrid KG+embedding retrieval outperform each method alone?
• RQ4 Does rebalancing via a KG (recency + impact Pareto) improve discovery of recent, influential docs?

Key findings

• KG IR outperforms ES IR on precision@K and recall@K for the target biomedical question across diseases.
• KG IR retrieves a broader, more diverse set of documents (long-tail) than ES IR, evidenced by more distinct clusters and wider graph-based retrieval regions.
• A hybrid ES IR + KG IR approach yields superior performance for small K and remains advantageous as K grows.
• EmbeddingIR tends to oversample near-question regions due to data imbalance, while KG IR balances data via graph structure and recency/impact prioritization.
• Combining KG and embedding signals yields a notable uplift over either method alone for early retrieval stages (K<100).