[Paper Review] Data Programming: Creating Large Training Sets, Quickly

Alexander Ratner, Christopher De+3 more|arXiv (Cornell University)|May 25, 2016

Machine Learning and Data Classification29 references359 citations

TL;DR

This paper introduces data programming, a framework where weak supervision signals via labeling functions are combined through a generative model to denoise labels and train discriminative models with noise-aware loss, achieving supervised-like learning rates with far fewer manually labeled examples.

ABSTRACT

Large labeled training sets are the critical building blocks of supervised learning methods and are key enablers of deep learning techniques. For some applications, creating labeled training sets is the most time-consuming and expensive part of applying machine learning. We therefore propose a paradigm for the programmatic creation of training sets called data programming in which users express weak supervision strategies or domain heuristics as labeling functions, which are programs that label subsets of the data, but that are noisy and may conflict. We show that by explicitly representing this training set labeling process as a generative model, we can "denoise" the generated training set, and establish theoretically that we can recover the parameters of these generative models in a handful of settings. We then show how to modify a discriminative loss function to make it noise-aware, and demonstrate our method over a range of discriminative models including logistic regression and LSTMs. Experimentally, on the 2014 TAC-KBP Slot Filling challenge, we show that data programming would have led to a new winning score, and also show that applying data programming to an LSTM model leads to a TAC-KBP score almost 6 F1 points over a state-of-the-art LSTM baseline (and into second place in the competition). Additionally, in initial user studies we observed that data programming may be an easier way for non-experts to create machine learning models when training data is limited or unavailable.

Motivation & Objective

• Motivate the high cost of hand-labeled datasets and the need for scalable weak supervision.
• Propose labeling functions as programmable weak supervision sources to generate large training sets.
• Model the labeling process as a generative/factor-graph framework to denoise labels and learn function accuracies and dependencies.
• Show that with O(1) labeling functions and O(ε^{-2}) unlabeled data, learning performance matches supervised methods under certain conditions.
• Demonstrate improved performance on real-world relation extraction tasks and report usability insights for domain experts.

Proposed method

• Define labeling functions λ: X → {-1,0,1} as weak supervision signals.
• Model the joint distribution of labeling outputs Λ and true label Y using a generative model μ_{α,β} with parameters α (accuracy) and β (coverage).
• Estimate α, β by maximum likelihood from unlabeled data via SGD (Eq. 2).
• Train a noise-aware discriminative model by minimizing a modified logistic loss L_{α̂, β̂} that conditions on observed Λ (Eq. 3).
• Provide theoretical guarantees: with m labeling functions and |S| unlabeled samples, E[‖α̂−α*‖²], E[‖β̂−β*‖²], and E[L(ŵ)−min_w L(w)] scale favorably, matching supervised-rate sample complexity under certain conditions.
• Extend the model to handle labeling-function dependencies with a dependency graph using a factor-graph representation, learning an augmented parameter θ (Eq. 7–9).
• Use Gibbs sampling and SGD for dependent cases and establish similar learning-rate guarantees as the independent case.

Experimental results

Research questions

• RQ1 Can a small set of user-defined labeling functions (weak supervision signals) generate training labels that, when denoised via a probabilistic model, yield discriminative models with competitive performance to fully supervised methods?
• RQ2 How does incorporating dependencies among labeling functions affect parameter learning and predictive performance?
• RQ3 What are the theoretical sample complexity and generalization guarantees for data programming compared to traditional supervised learning?
• RQ4 Does data programming scale to real-world NLP tasks and work synergistically with automated feature generation (e.g., LSTMs)?

Key findings

• Data programming can denoise weak supervision signals by learning labeling-function accuracies and dependencies.
• Under specified conditions, the framework achieves the same asymptotic learning rate as supervised methods, with O(1) labeling functions and Õ(ε^{-2}) unlabeled data.
• Applying data programming to real-world relation extraction tasks yielded average F1 improvements (e.g., +2.34 points) over distant supervision baselines and nearly +6 F1 points over a state-of-the-art LSTM baseline in TAC-KBP 2014 benchmarks.
• Using LSTM-generated features with data programming produced large gains, including a 9.79-point precision improvement and 3.12-point F1 gain in TAC-KBP (News).
• A usability study with bioinformatics researchers suggested labeling-function development can be faster and accessible to non-experts, enabling competitive models with limited or no labeled data.