[Paper Review] Learning Conceptual-Contextual Embeddings for Medical Text

Xiao Zhang, Dejing Dou+1 more|arXiv (Cornell University)|Aug 16, 2019

Topic Modeling41 references4 citations

TL;DR

This paper introduces Conceptual-Contextual (CC) embeddings, a contextual text representation model that integrates structured knowledge from medical knowledge graphs into neural text representations using a multi-layer bi-directional LSTM encoder. Trained on the UMLS knowledge base, CC embeddings significantly improve performance on medical NLP tasks such as readmission and mortality prediction in EHRs and medical natural language inference, outperforming strong baselines and state-of-the-art models in some cases.

ABSTRACT

External knowledge is often useful for natural language understanding tasks. We introduce a contextual text representation model called Conceptual-Contextual (CC) embeddings, which incorporates structured knowledge into text representations. Unlike entity embedding methods, our approach encodes a knowledge graph into a context model. CC embeddings can be easily reused for a wide range of tasks just like pre-trained language models. Our model effectively encodes the huge UMLS database by leveraging semantic generalizability. Experiments on electronic health records (EHRs) and medical text processing benchmarks showed our model gives a major boost to the performance of supervised medical NLP tasks.

Motivation & Objective

• To address the challenge of incorporating structured medical knowledge into neural text representation models for improved NLP performance.
• To develop a method that encodes knowledge graph relationships directly into contextual embeddings, avoiding alignment issues between text and entity representations.
• To enable reusable, concept-aware representations that generalize across diverse medical NLP tasks without task-specific retraining.
• To validate the effectiveness of CC embeddings on real-world medical text, including EHRs and clinical inference benchmarks.
• To bridge the gap between knowledge graph embeddings and pre-trained contextual language models in the medical domain.

Proposed method

• The CC embedding model uses a multi-layer bi-directional LSTM encoder to generate contextual representations of concept mentions in text.
• The model is trained on a knowledge graph embedding objective, learning to produce concept representations that satisfy approximate translational relationships (e.g., Encode('Cortisone') + may_prevent ≈ Encode('Rheumatoid Arthritis'))
• Concept mentions in text are encoded using the trained encoder, which absorbs structured knowledge from the UMLS knowledge base during training.
• The model is pre-trained on a large corpus of medical text to recognize and encode medical concepts and their relationships.
• After pre-training, the model can be fine-tuned for downstream tasks or used as a feature extractor with a pooling and linear layer.
• The approach distinguishes between language-inferable and non-language-inferable knowledge, focusing on capturing semantic relationships from textual descriptions in the knowledge base.

Experimental results

Research questions

• RQ1 Can structured medical knowledge from knowledge graphs be effectively integrated into contextual text representations to improve NLP performance?
• RQ2 Does a knowledge-aware text representation model outperform standard contextual models like BERT or LSTM in medical text understanding tasks?
• RQ3 To what extent can concept-aware embeddings improve downstream tasks such as ICU readmission and mortality prediction in EHRs?
• RQ4 Can CC embeddings enhance performance in medical natural language inference tasks compared to standard models?
• RQ5 How does the model handle non-language-inferable knowledge, such as brand-name relations, compared to purely linguistic relationships?

Key findings

• The CC-LSTM model achieved an AUC of 0.839 on 30-day post-discharge mortality prediction, outperforming a standard LSTM baseline (0.823) and surpassing state-of-the-art results that use additional clinical data.
• On 30-day readmission prediction, the CC model achieved a higher AUC than the baseline LSTM and matched or exceeded state-of-the-art models despite using only textual discharge summaries.
• In the MedNLI benchmark, the CC-ESIM model achieved 75.2% test accuracy, a significant improvement over the baseline ESIM model (71.3%), demonstrating the benefit of external knowledge in inference tasks.
• The performance gain in all tasks indicates that CC embeddings successfully inject structured medical knowledge into contextual representations, enhancing generalization and robustness.
• The model’s ability to encode both language-inferable and non-language-inferable knowledge suggests it captures diverse semantic relationships from textual descriptions in medical KBs.
• The results validate that knowledge graph integration via a context-aware encoder is effective and scalable, especially in low-resource or knowledge-intensive NLP scenarios.