[Paper Review] A hybrid deep learning approach for medical relation extraction

Veera Raghavendra Chikka, Kamalakar Karlapalem|arXiv (Cornell University)|Jun 26, 2018

Topic Modeling10 references10 citations

TL;DR

This paper proposes a hybrid deep learning approach combining bidirectional LSTM with rule-based modeling to improve medical relation extraction between treatments and medical problems. By leveraging both word-level and sentence-level features—including part-of-speech tags, position vectors, and pointwise mutual information—the model achieves an F-score of 0.52 on the I2b2 2010 benchmark, with significant gains in TrIP and TrNAP relations through integration of high-precision rule-based outputs.

ABSTRACT

Mining relationships between treatment(s) and medical problem(s) is vital in the biomedical domain. This helps in various applications, such as decision support system, safety surveillance, and new treatment discovery. We propose a deep learning approach that utilizes both word level and sentence-level representations to extract the relationships between treatment and problem. While deep learning techniques demand a large amount of data for training, we make use of a rule-based system particularly for relationship classes with fewer samples. Our final relations are derived by jointly combining the results from deep learning and rule-based models. Our system achieved a promising performance on the relationship classes of I2b2 2010 relation extraction task.

Motivation & Objective

• To address the challenge of extracting relationships between treatments and medical problems in clinical text, which is critical for decision support and drug safety surveillance.
• To improve deep learning performance in low-resource relation classes by integrating rule-based systems for underrepresented relationship types.
• To explore the effectiveness of bidirectional LSTM over traditional CNNs and SVMs in capturing sequential dependencies for relation classification.
• To evaluate the impact of sentence-level representations—such as POS sequences and mutual information—on relation extraction accuracy.

Proposed method

• A bidirectional LSTM network processes input sentences with word-level features (tokens, POS tags, chunk tags) and position vectors indicating distance to treatment and problem entities.
• Sentence-level features include POS tag sequences (top 100 frequent sequences), pointwise mutual information for entity co-occurrence, and assertion keywords from cTAKES.
• Word embeddings are initialized randomly and fine-tuned via backpropagation during training, with input features concatenated into a dense matrix representation.
• A rule-based system is developed for relation types with low training sample counts (e.g., TrIP, TrNAP), using linguistic patterns and semantic cues such as verbs like 'controlled' or 'resistant'.
• The final predictions are generated by merging high-precision rule-based outputs with Bi-LSTM predictions, particularly enhancing performance on low-resource relations.
• The model is trained using cross-entropy loss and optimized via Adam, with hyperparameters tuned on validation data.

Experimental results

Research questions

• RQ1 Can bidirectional LSTM outperform traditional SVM and CNN-based models in capturing sequential dependencies for medical relation extraction?
• RQ2 How do sentence-level features such as POS sequences and mutual information contribute to relation classification beyond word-level representations?
• RQ3 To what extent can rule-based systems improve performance on low-resource relation types (e.g., TrIP, TrNAP) when combined with deep learning?
• RQ4 Does joint inference from Bi-LSTM and rule-based models yield better F-score than either method alone on the I2b2 2010 benchmark?

Key findings

• The Bi-LSTM model achieved an F-score of 0.58 on the TrAP relation, outperforming SVM and demonstrating the benefit of large-scale training data for deep learning.
• The rule-based system achieved high precision on TrIP and TrNAP relations, with 0.47 and 0.30 F-scores respectively, and significantly improved the Bi-LSTM performance when combined.
• The hybrid Bi-LSTM and rule-based system achieved a total F-score of 0.52, representing a 0.01 improvement over the Bi-LSTM alone, with notable gains in TrIP (0.42 vs. 0.41) and TrNAP (0.30 vs. 0.22).
• The integration of rule-based outputs was most effective for relations with fewer training samples, such as TrIP and TrNAP, which were underperforming in the Bi-LSTM model alone.
• The study found that ambiguous annotations in the gold standard, such as 'treated for' being labeled as TrAP or TrIP, contributed to performance variance and highlighted challenges in dataset quality.
• The use of full-sentence representations with position-aware embeddings and sentence-level features improved context modeling compared to local window-based approaches.