[Paper Review] SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data

Jason Fries, Sen Wu+2 more|arXiv (Cornell University)|Apr 20, 2017

Topic ModelingComputer Science16 references90 citations

TL;DR

SwellShark builds biomedical NER taggers from weak supervision using a multinomial generative model to denoise signals from lexicons and heuristics, achieving competitive results without hand-labeled data.

ABSTRACT

We present SwellShark, a framework for building biomedical named entity recognition (NER) systems quickly and without hand-labeled data. Our approach views biomedical resources like lexicons as function primitives for autogenerating weak supervision. We then use a generative model to unify and denoise this supervision and construct large-scale, probabilistically labeled datasets for training high-accuracy NER taggers. In three biomedical NER tasks, SwellShark achieves competitive scores with state-of-the-art supervised benchmarks using no hand-labeled training data. In a drug name extraction task using patient medical records, one domain expert using SwellShark achieved within 5.1% of a crowdsourced annotation approach -- which originally utilized 20 teams over the course of several weeks -- in 24 hours.

Motivation & Objective

• Reduce or eliminate the need for hand-labeled training data in biomedical NER
• Leverage biomedical lexicons, ontologies, and heuristics as weak supervision sources
• Automatically generate candidates and labeling functions to assemble large probabilistic training sets
• Unify and denoise weak supervision with a multinomial generative model to infer entity boundaries
• Demonstrate scalability and rapid domain adaptation across multiple biomedical NER tasks

Proposed method

• Automatic candidate generation to identify potential entity mentions in documents.
• Autogenerated supervision by transforming lexicons and heuristics into labeling functions.
• Labeling function generators that convert supervision resources into multiple labeling functions.
• A multinomial generative model that accounts for overlapping spans and mutual exclusion constraints to estimate entity boundaries.
• Sampling-based data construction that creates probabilistic labels for training discriminative models (CRF/LSTM-CRF).
• Scalability experiments showing performance with up to 100k unlabeled documents.

Experimental results

Research questions

• RQ1 How well do automated candidate generation strategies perform compared to hand-tuned generators?
• RQ2 How effective is autogenerated supervision derived from lexical resources for biomedical NER?
• RQ3 Can SwellShark train high-accuracy taggers using large unlabeled datasets with minimal human input?
• RQ4 How close can weakly supervised models get to supervised baselines across Disease and Chemical NER tasks?
• RQ5 How quickly can a domain-specific NER tagger be built in a constrained time (e.g., 24 hours) using SwellShark?

Key findings

• SwellShark achieves competitive scores with state-of-the-art supervised benchmarks using no hand-labeled training data.
• With up to 100k unlabeled documents, tagger accuracy improves by up to 6.7% (4.9 F1 points) over the same models trained on ≤1k documents.
• In a drug name extraction task, a single domain expert built a tagger in 24 hours that scored within 5.1% of a crowdsourced annotation approach that used 20 teams over weeks.
• Automatic noun-phrase candidate generation can approach hand-tuned performance, and LSTM-CRF models benefit from embeddings, generally converging faster with larger unlabeled data.
• Modeling spanset dependencies (mutual exclusion) improves F1 by up to 4.7% (3.4 points) in some setups.