[Paper Review] Applying deep learning techniques on medical corpora from the World Wide Web: a prototypical system and evaluation

José Antonio Miñarro-Giménez, Oscar Marín-Alonso+1 more|arXiv (Cornell University)|Feb 12, 2015

Biomedical Text Mining and Ontologies7 references16 citations

TL;DR

This study evaluates word2vec for mining pharmaceutical relationships from unstructured web-based medical text, using the NDF-RT ontology as a gold standard. Despite efficient vector learning, word2vec achieved only 49.28% accuracy in identifying relationships like 'may treat' or 'has physiological effect,' indicating limited performance for automatic knowledge base population despite potential as a curation starting point.

ABSTRACT

BACKGROUND: The amount of biomedical literature is rapidly growing and it is becoming increasingly difficult to keep manually curated knowledge bases and ontologies up-to-date. In this study we applied the word2vec deep learning toolkit to medical corpora to test its potential for identifying relationships from unstructured text. We evaluated the efficiency of word2vec in identifying properties of pharmaceuticals based on mid-sized, unstructured medical text corpora available on the web. Properties included relationships to diseases ('may treat') or physiological processes ('has physiological effect'). We compared the relationships identified by word2vec with manually curated information from the National Drug File - Reference Terminology (NDF-RT) ontology as a gold standard. RESULTS: Our results revealed a maximum accuracy of 49.28% which suggests a limited ability of word2vec to capture linguistic regularities on the collected medical corpora compared with other published results. We were able to document the influence of different parameter settings on result accuracy and found and unexpected trade-off between ranking quality and accuracy. Pre-processing corpora to reduce syntactic variability proved to be a good strategy for increasing the utility of the trained vector models. CONCLUSIONS: Word2vec is a very efficient implementation for computing vector representations and for its ability to identify relationships in textual data without any prior domain knowledge. We found that the ranking and retrieved results generated by word2vec were not of sufficient quality for automatic population of knowledge bases and ontologies, but could serve as a starting point for further manual curation.

Motivation & Objective

• To assess the feasibility of using deep learning, specifically word2vec, to extract biomedical relationships from unstructured web text.
• To evaluate word2vec’s performance in identifying drug-disease and drug-physiological effect relationships from mid-sized, real-world medical corpora.
• To compare word2vec-generated relationships against manually curated gold standard data from the NDF-RT ontology.
• To investigate the impact of preprocessing and hyperparameter tuning on model accuracy and ranking quality.
• To determine whether word2vec outputs can serve as a viable foundation for automated curation of medical knowledge bases.

Proposed method

• The authors applied the word2vec model to a corpus of unstructured medical text collected from the World Wide Web.
• Relationships such as 'may treat' and 'has physiological effect' were identified using vector similarity in the learned embedding space.
• Model performance was evaluated by comparing predicted relationships against the NDF-RT ontology as a gold standard.
• Preprocessing strategies, including normalization to reduce syntactic variability, were applied to improve model utility.
• Hyperparameters such as vector size, window size, and learning rate were systematically varied to assess their impact on accuracy and ranking.
• A cosine similarity-based approach was used to retrieve related drug-term pairs, with results ranked and evaluated for precision.

Experimental results

Research questions

• RQ1 Can word2vec effectively identify drug-disease and drug-physiological effect relationships from unstructured medical web text?
• RQ2 How does model accuracy vary with different hyperparameter settings in the word2vec framework?
• RQ3 What is the impact of syntactic preprocessing on the quality of learned vector representations and downstream relationship extraction?
• RQ4 Is there a trade-off between ranking quality and accuracy in word2vec-based relationship retrieval?
• RQ5 To what extent can word2vec outputs support the automatic population of medical knowledge bases and ontologies?

Key findings

• The maximum accuracy achieved by word2vec in identifying drug relationships was 49.28%, indicating limited effectiveness for this task.
• A significant trade-off was observed between ranking quality and accuracy, where higher-ranked results were not necessarily more accurate.
• Preprocessing the corpus to reduce syntactic variability significantly improved the utility of the trained vector models.
• The results suggest that word2vec alone is insufficient for reliable automatic curation of medical knowledge bases.
• Despite low accuracy, word2vec remains an efficient tool for generating vector representations without prior domain knowledge.
• The study confirms that word2vec outputs can serve as a useful starting point for subsequent manual curation efforts.