[Paper Review] A Relation Extraction Approach for Clinical Decision Support

Maristella Agosti, Giorgio Maria Di Nunzio+2 more|arXiv (Cornell University)|May 3, 2019

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology22 references3 citations

TL;DR

This paper proposes using semantic relations between medical concepts—extracted via rule-based and learning-based methods—as indexing and retrieval features in clinical decision support systems. It demonstrates that relation-based retrieval significantly improves precision over baseline methods on specific queries, with a 20% average nDCG improvement when relations are informative, though performance is inconsistent due to query length and structural mismatches.

ABSTRACT

In this paper, we investigate how semantic relations between concepts extracted from medical documents can be employed to improve the retrieval of medical literature. Semantic relations explicitly represent relatedness between concepts and carry high informative power that can be leveraged to improve the effectiveness of retrieval functionalities of clinical decision support systems. We present preliminary results and show how relations are able to provide a sizable increase of the precision for several topics, albeit having no impact on others. We then discuss some future directions to minimize the impact of negative results while maximizing the impact of good results.

Motivation & Objective

• To investigate whether semantic relations between medical concepts can enhance retrieval effectiveness in clinical decision support systems.
• To compare rule-based and learning-based relation extraction methods for improving precision in case-based medical literature retrieval.
• To assess the impact of relation-based indexing on retrieval performance using the OHSUMED dataset.
• To identify conditions under which relations improve retrieval and address cases where they yield no useful results.
• To explore future directions for relation extraction that better align with IR needs in clinical settings.

Proposed method

• A rule-based method assigns relations to concept pairs within the same sentence if they exist in a reference knowledge base.
• A learning-based method trains a sentence-level relation extractor to infer relations based on contextual cues in medical text.
• Relations extracted from queries and documents are used to compute passage-level scores via a weighted sum: $\text{score}(q,d) = \sum_{p\in d} \frac{|R_p \cap R_q|}{|R_q|} \cdot \text{BM25}(p,q)$.
• Documents are scored based on the number of relevant relations they share with the query, using BM25 for passage retrieval.
• The approach evaluates performance using nDCG on the OHSUMED collection, comparing BoW, BoC, and BoR (bag of relations) representations.
• Experiments are restricted to queries where relations can be extracted, focusing on cases with non-zero relevance scores.

Experimental results

Research questions

• RQ1 Can semantic relations extracted from medical documents and queries improve the precision of clinical decision support retrieval systems?
• RQ2 How do rule-based and learning-based relation extraction methods compare in effectiveness for clinical information retrieval?
• RQ3 In what conditions do relations significantly enhance retrieval performance, and when do they fail?
• RQ4 To what extent do relations outperform traditional concept-based or term-based retrieval in case-based medical literature search?
• RQ5 How can relation extraction be optimized to reduce false negatives and improve consistency in low-resource query scenarios?

Key findings

• The rule-based method successfully extracted relations from 44 out of 106 OHSUMED queries, with 39 queries yielding matching document relations.
• For queries where relations provided non-zero results, the average nDCG improvement over concept-based retrieval was 20%, and this improvement was statistically significant.
• The learning-based method extracted relations from only 25 queries, with 12 yielding matching documents, indicating lower coverage due to query structure and syntactic mismatch.
• In cases where relations were informative, they outperformed baseline term-based retrieval, but no significant difference was found between relations and terms overall due to inconsistent results.
• The variance in nDCG scores—ranging from 0 to 1—highlights the high informative potential of relations when correctly extracted, but also their sensitivity to extraction quality and query length.
• The study identifies that short, non-narrative queries limit relation extraction, and future work should focus on IR-optimized relation extraction to overcome low recall in such settings.