[Paper Review] Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods

Daniel Y. Fu, Mayee F. Chen+4 more|arXiv (Cornell University)|Feb 27, 2020

Machine Learning and Data Classification56 references17 citations

TL;DR

This paper introduces FlyingSquid, a weak supervision framework that accelerates label model training by reformulating latent variable estimation as a series of closed-form triplet problems, achieving 170x faster parameter estimation than prior methods while matching or exceeding state-of-the-art performance on benchmark and video analysis tasks without requiring SGD tuning or iterative optimization.

ABSTRACT

Weak supervision is a popular method for building machine learning models without relying on ground truth annotations. Instead, it generates probabilistic training labels by estimating the accuracies of multiple noisy labeling sources (e.g., heuristics, crowd workers). Existing approaches use latent variable estimation to model the noisy sources, but these methods can be computationally expensive, scaling superlinearly in the data. In this work, we show that, for a class of latent variable models highly applicable to weak supervision, we can find a closed-form solution to model parameters, obviating the need for iterative solutions like stochastic gradient descent (SGD). We use this insight to build FlyingSquid, a weak supervision framework that runs orders of magnitude faster than previous weak supervision approaches and requires fewer assumptions. In particular, we prove bounds on generalization error without assuming that the latent variable model can exactly parameterize the underlying data distribution. Empirically, we validate FlyingSquid on benchmark weak supervision datasets and find that it achieves the same or higher quality compared to previous approaches without the need to tune an SGD procedure, recovers model parameters 170 times faster on average, and enables new video analysis and online learning applications.

Motivation & Objective

• To address the computational inefficiency of existing weak supervision methods that rely on iterative optimization like SGD or Gibbs sampling.
• To enable fast, scalable weak supervision for complex, high-dimensional data such as video and streaming data with temporal dependencies.
• To develop a method that achieves closed-form solutions for latent variable model parameters in weak supervision, eliminating the need for iterative hyperparameter tuning.
• To prove generalization error bounds under model misspecification and establish sample optimality via information-theoretic lower bounds.
• To enable online learning with window-based parameter updates by leveraging the speed of closed-form triplet estimation.

Proposed method

• The method decomposes the latent variable model estimation into minimal subproblems by forming triplets of labeling sources, reducing the problem to solving systems of equations with closed-form solutions.
• For a class of binary Ising models, the framework derives a system of linear equations from triplet agreements and disagreements, which are solved directly without iterative optimization.
• It uses median or mean aggregation across all possible triplets to improve robustness, especially in high-abstention scenarios where some sources label no examples.
• The framework integrates parameter estimation into the training loop of downstream discriminative models, enabling end-to-end joint learning.
• It supports online learning by maintaining a sliding window of data and re-estimating parameters using closed-form triplet solutions at each step.
• The approach is validated on benchmark datasets and video analysis tasks, demonstrating speed and accuracy advantages.

Experimental results

Research questions

• RQ1 Can we achieve closed-form solutions for latent variable model parameters in weak supervision, eliminating the need for iterative optimization like SGD?
• RQ2 How does the generalization error of models trained with FlyingSquid compare to fully supervised learning under model misspecification?
• RQ3 Can the framework scale to high-dimensional, temporally dependent data such as video sequences, where prior methods are too slow?
• RQ4 Does the use of triplet-based aggregation improve robustness to noisy or abstaining labeling sources compared to single-triplet selection?
• RQ5 Can the framework support online learning with low-latency parameter updates in streaming settings?

Key findings

• FlyingSquid achieves 170 times faster parameter estimation on average compared to prior weak supervision methods, with no need for SGD hyperparameter tuning.
• On benchmark text classification tasks, FlyingSquid matches or exceeds previous state-of-the-art performance, improving F1 scores by up to 4.9 points.
• The method maintains strong performance even with high-abstention rates by using mean aggregation, which is more robust than median when many triplets yield zero accuracy scores.
• Abllation studies show that random replacement of abstentions with random votes severely degrades performance, while single-triplet selection leads to high variance and instability.
• FlyingSquid enables new applications in video analysis and online learning by allowing real-time parameter updates within the training loop.
• Theoretical analysis confirms that generalization error scales at the same asymptotic rate as in supervised learning, and the method is sample-optimal up to constant factors.