[Paper Review] Concept Extraction to Identify Adverse Drug Reactions in Medical Forums: A Comparison of Algorithms

Alejandro Metke‐Jimenez, Sarvnaz Karimi|arXiv (Cornell University)|Apr 27, 2015

Biomedical Text Mining and Ontologies38 references19 citations

TL;DR

This study evaluates multiple concept extraction methods—ranging from dictionary-based approaches to a novel CRF-based machine learning model—for identifying adverse drug reactions (ADRs) and drugs in patient-generated medical forum text. The proposed CRF method with Ontoserver normalization significantly outperformed MetaMap and other techniques, especially in low-resource settings, demonstrating superior precision and F1 scores on the Cadec corpus.

ABSTRACT

Social media is becoming an increasingly important source of information to complement traditional pharmacovigilance methods. In order to identify signals of potential adverse drug reactions, it is necessary to first identify medical concepts in the social media text. Most of the existing studies use dictionary-based methods which are not evaluated independently from the overall signal detection task. We compare different approaches to automatically identify and normalise medical concepts in consumer reviews in medical forums. Specifically, we implement several dictionary-based methods popular in the relevant literature, as well as a method we suggest based on a state-of-the-art machine learning method for entity recognition. MetaMap, a popular biomedical concept extraction tool, is used as a baseline. Our evaluations were performed in a controlled setting on a common corpus which is a collection of medical forum posts annotated with concepts and linked to controlled vocabularies such as MedDRA and SNOMED CT. To our knowledge, our study is the first to systematically examine the effect of popular concept extraction methods in the area of signal detection for adverse reactions. We show that the choice of algorithm or controlled vocabulary has a significant impact on concept extraction, which will impact the overall signal detection process. We also show that our proposed machine learning approach significantly outperforms all the other methods in identification of both adverse reactions and drugs, even when trained with a relatively small set of annotated text.

Motivation & Objective

• To systematically evaluate and compare concept extraction methods for adverse drug reaction (ADR) detection in social media text.
• To assess the impact of algorithm choice and controlled vocabulary (e.g., MedDRA, SNOMED CT, CHV) on ADR signal detection performance.
• To evaluate the effectiveness of a novel CRF-based machine learning approach against established dictionary-based and MetaMap methods in a controlled, gold-standard setting.
• To provide a benchmark using the Cadec corpus for future research in ADR mining from patient forums.

Proposed method

• The study implements and compares multiple concept extraction techniques: dictionary-based methods using UMLS, AMT, and CHV; MetaMap; and a CRF-based model trained on a small annotated dataset.
• Concept normalization is performed using Ontoserver, UMLS, AMT, and CHV, with results evaluated against gold-standard annotations from the Cadec corpus.
• A multi-level evaluation framework is used, including strict span matching, relaxed matching (overlap-based), and semantic distance-aware normalization evaluation.
• Statistical significance testing is applied to assess performance differences across methods, particularly focusing on F1, precision, and recall.
• The CRF model uses features from the text span and surrounding context, with normalization decisions informed by ontology structure.
• Evaluation metrics include exact span match, partial overlap (relaxed match), and semantic similarity to assess normalization accuracy.

Experimental results

Research questions

• RQ1 How do different concept extraction algorithms perform in identifying adverse drug reactions and drugs in patient forum text?
• RQ2 What is the impact of controlled vocabulary choice (e.g., MedDRA, SNOMED CT, CHV) on concept extraction performance?
• RQ3 Does a machine learning-based approach (CRF) outperform traditional dictionary-based methods and MetaMap in ADR concept extraction?
• RQ4 How do different evaluation strategies—strict, relaxed, and semantic distance—alter the perceived performance of concept extraction systems?

Key findings

• The CRF-based model with Ontoserver normalization achieved the highest F1 score of 0.867 for concept identification, significantly outperforming MetaMap (F1 = 0.814) and all dictionary-based methods.
• The CRF model achieved a precision of 0.988 and recall of 0.773, indicating strong performance even with limited training data.
• Dictionary-based methods using CHV or UMLS showed poor performance, with F1 scores of 0.000 for several configurations, highlighting the limitations of lexicon-based approaches in noisy, informal text.
• MetaMap performed poorly overall, with F1 scores of 0.814 across all tasks, and failed to identify many drug and ADR mentions due to case sensitivity and spelling variations.
• Relaxed matching criteria were found to be too lenient, as single-token overlaps could constitute a match, suggesting the need for threshold-based overlap metrics.
• The study demonstrates that concept normalization performance can be improved by incorporating semantic distance (e.g., parent-child relationships in ontologies), as current methods treat all mismatches as complete failures.