[Paper Review] ITTC @ TREC 2021 Clinical Trials Track

Thinh Hung Truong, Yulia Otmakhova+7 more|arXiv (Cornell University)|Feb 16, 2022

Biomedical Text Mining and Ontologies4 citations

TL;DR

This paper presents the ITTC system for the TREC 2021 Clinical Trials Track, which uses multiple text representation strategies—complete text, keywords, and entity-based queries—combined with BM25 retrieval to match patient admission notes to eligible clinical trials. The best-performing run (Run 5) achieved an AP@10 of 0.273, significantly above the median, demonstrating that negation-aware entity filtering improves retrieval performance over standard keyword or full-text methods.

ABSTRACT

This paper describes the submissions of the Natural Language Processing (NLP) team from the Australian Research Council Industrial Transformation Training Centre (ITTC) for Cognitive Computing in Medical Technologies to the TREC 2021 Clinical Trials Track. The task focuses on the problem of matching eligible clinical trials to topics constituting a summary of a patient's admission notes. We explore different ways of representing trials and topics using NLP techniques, and then use a common retrieval model to generate the ranked list of relevant trials for each topic. The results from all our submitted runs are well above the median scores for all topics, but there is still plenty of scope for improvement.

Motivation & Objective

• To develop an NLP-based system that improves automated matching of clinical trials to patient admission notes.
• To investigate how different text representations (full text, keywords, entities) affect retrieval performance in patient-trial matching.
• To evaluate strategies for handling inclusion and exclusion criteria, particularly negation in clinical trial criteria.
• To address challenges in clinical text such as abbreviations, implicit diagnoses, and noisy information in patient summaries.
• To improve recruitment efficiency in clinical trials by identifying eligible patients through semantic and structural text analysis.

Proposed method

• Used BM25 as the base retrieval model for ranking clinical trials based on topic queries.
• Explored multiple query representations: full text, keyword-extracted terms, and entity-based strings from patient notes.
• Applied negation detection to exclude negated entities from query representation, improving specificity.
• Treated clinical trial documents by separating inclusion and exclusion criteria, then used set difference to refine results.
• Combined metadata (age, gender, clinical variables) with text representations to enhance relevance.
• Evaluated performance using AP@10, comparing runs against baseline and median scores across all teams.

Experimental results

Research questions

• RQ1 How do different text representation methods (full text, keywords, entities) impact retrieval performance in patient-trial matching?
• RQ2 Can negation-aware query construction improve the identification of eligible clinical trials?
• RQ3 How effective is the set-difference approach in combining inclusion and exclusion criteria for filtering relevant trials?
• RQ4 Does incorporating structured metadata (e.g., age, gender) improve retrieval performance compared to pure text matching?
• RQ5 What are the main challenges in clinical text that lead to poor performance on specific topics?

Key findings

• The best-performing run (Run 5: entity-focused with negation) achieved an AP@10 of 0.273, which was 0.111 points above the median score across all teams.
• All submitted runs outperformed the baseline (AP@10 = 0.216), with Run 3 (set difference of inclusion and exclusion texts) achieving the highest median improvement (0.127 points).
• Runs using keyword-based query representation performed worse than full-text or entity-based methods, indicating information loss during keyword extraction.
• Excluding negated entities from the query (Run 5) proved more effective than reversing keyword polarity (Run 2), suggesting better handling of negation semantics.
• Topics with high abbreviation density (e.g., Topics 9, 10, 32, 53) or low medical term frequency (e.g., Topic 44) were consistently harder to match, often due to few eligible trials in the corpus.
• Metadata such as age, gender, and clinical variables improved retrieval when combined with text, as shown by Run 2’s better performance over the full-text baseline.