[Paper Review] A comparative study of approaches in user-centered health information retrieval

Harsh Thakkar, Ganesh Iyer+1 more|arXiv (Cornell University)|May 7, 2015

Biomedical Text Mining and Ontologies19 references4 citations

TL;DR

This paper evaluates user-centered biomedical information retrieval systems in the CLEF eHealth 2014 Task 3, comparing Language Model (LM) and Vector Space Model (VSM) approaches. It finds that LM-based systems significantly outperform VSM-based systems, achieving a state-of-the-art MAP of 0.4146, P@10 of 0.7560, and NDCG@10 of 0.7445 using medical ontologies like MeSH and UMLS.

ABSTRACT

In this paper, we survey various user-centered or context-based biomedical health information retrieval systems. We present and discuss the performance of systems submitted in CLEF eHealth 2014 Task 3 for this purpose. We classify and focus on comparing the two most prevalent retrieval models in biomedical information retrieval namely: Language Model (LM) and Vector Space Model (VSM). We also report on the effectiveness of using external medical resources and ontologies like MeSH, Metamap, UMLS, etc. We observed that the L.M. based retrieval systems outperform VSM based systems on various fronts. From the results we conclude that the state-of-art system scores for MAP was 0.4146, P@10 was 0.7560 and NDCG@10 was 0.7445, respectively. All of these score were reported by systems built on language modelling approaches.

Motivation & Objective

• To evaluate and compare user-centered biomedical information retrieval systems in the CLEF eHealth 2014 Task 3.
• To analyze the effectiveness of Language Model (LM) and Vector Space Model (VSM) approaches in health information retrieval.
• To assess the impact of external medical resources and ontologies such as MeSH, Metamap, and UMLS on retrieval performance.
• To identify the state-of-the-art performance metrics in context-aware health information retrieval.

Proposed method

• The study analyzes systems submitted to the CLEF eHealth 2014 Task 3, focusing on user-centered and context-based retrieval approaches.
• It classifies and compares two dominant retrieval models: Language Model (LM) and Vector Space Model (VSM).
• External medical resources including MeSH, Metamap, and UMLS are evaluated for their contribution to retrieval effectiveness.
• Performance is measured using standard IR metrics: Mean Average Precision (MAP), Precision at 10 (P@10), and Normalized Discounted Cumulative Gain at 10 (NDCG@10).
• The analysis focuses on how ontology integration enhances query understanding and result relevance in biomedical search.

Experimental results

Research questions

• RQ1 How do Language Model (LM) and Vector Space Model (VSM) approaches compare in performance on user-centered health information retrieval tasks?
• RQ2 To what extent do external medical ontologies such as MeSH, UMLS, and Metamap improve retrieval effectiveness in biomedical search?
• RQ3 What are the best-performing metrics (MAP, P@10, NDCG@10) achieved by state-of-the-art systems in the CLEF eHealth 2014 Task 3?
• RQ4 Which retrieval model architecture yields the highest precision and normalized gain in ranked results for health queries?

Key findings

• Language Model (LM)-based systems outperformed Vector Space Model (VSM)-based systems across all evaluation metrics.
• The best-performing system achieved a Mean Average Precision (MAP) of 0.4146, indicating strong average precision across multiple queries.
• The top system recorded a Precision at 10 (P@10) of 0.7560, reflecting high relevance of the top 10 retrieved results.
• Normalized Discounted Cumulative Gain at 10 (NDCG@10) reached 0.7445, showing strong ranking quality of the top results.
• The integration of medical ontologies such as MeSH, UMLS, and Metamap contributed significantly to improved retrieval performance.
• The study confirms that LM-based models are currently the state-of-the-art approach for user-centered health information retrieval.