[Paper Review] WRENCH: A Comprehensive Benchmark for Weak Supervision

Jieyu Zhang, Yue Yu+5 more|arXiv (Cornell University)|Sep 23, 2021

Machine Learning and Data Classification104 references39 citations

TL;DR

WRENCH provides a standardized benchmark platform with 22 real-world datasets, various weak supervision sources (real, synthetic, procedural), and a modular framework to evaluate weak supervision methods, enabling over 120 method variants across classification and sequence tagging.

ABSTRACT

Recent Weak Supervision (WS) approaches have had widespread success in easing the bottleneck of labeling training data for machine learning by synthesizing labels from multiple potentially noisy supervision sources. However, proper measurement and analysis of these approaches remain a challenge. First, datasets used in existing works are often private and/or custom, limiting standardization. Second, WS datasets with the same name and base data often vary in terms of the labels and weak supervision sources used, a significant "hidden" source of evaluation variance. Finally, WS studies often diverge in terms of the evaluation protocol and ablations used. To address these problems, we introduce a benchmark platform, WRENCH, for thorough and standardized evaluation of WS approaches. It consists of 22 varied real-world datasets for classification and sequence tagging; a range of real, synthetic, and procedurally-generated weak supervision sources; and a modular, extensible framework for WS evaluation, including implementations for popular WS methods. We use WRENCH to conduct extensive comparisons over more than 120 method variants to demonstrate its efficacy as a benchmark platform. The code is available at https://github.com/JieyuZ2/wrench.

Motivation & Objective

• Address the lack of standardized benchmarks in weak supervision (WS) by providing a diverse, public benchmark platform.
• Enable thorough evaluation of WS methods across datasets, sources, and evaluation protocols.
• Analyze how weak supervision attributes affect WS method performance using procedural and synthetic generators.
• Offer a modular codebase with standardized evaluation scripts and baselines to facilitate future comparisons.

Proposed method

• Introduce 22 real-world datasets for classification and sequence tagging with diverse domains and labeling functions (LFs).
• Provide procedural and synthetic LF generators to systematically explore LF properties (accuracy, propensity, correlation, data-dependency).
• Offer a unified, extensible Python framework with implementations of popular WS methods and standardized evaluation metrics.
• Enable over 100 method variants by combining label models, end models, and joint models with soft/hard labels.
• Present baseline methods for both classification and sequence tagging tasks (e.g., MV, DS, DP, MeTaL, FS, HMM, CHMM, ConNet, BERT variants).
• Demonstrate utility through extensive experiments across datasets to compare WS methods and ablations.

Experimental results

Research questions

• RQ1 How does standardizing WS benchmarks affect fair comparison across methods and datasets?
• RQ2 What is the impact of different weak supervision source properties (accuracy, propensity, correlation, data-dependency) on WS method performance?
• RQ3 How do two-stage (label model + end model) versus one-stage (joint) WS approaches perform across varied tasks and data domains?
• RQ4 To what extent do end-model choices (e.g., fine-tuned language models) influence WS outcomes compared to label models alone?
• RQ5 What guidance can be provided for choosing LF types and evaluation protocols to obtain robust WS results?

Key findings

• There is no single WS method that consistently outperforms others across all datasets, highlighting task- and LF-dependent performance.
• Fine-tuning large pre-trained language models often yields strong end-model performance on textual data, frequently surpassing label-model-only approaches.
• Soft labels tend to improve end-model performance over hard labels, especially as end models become deeper.
• LF quality, coverage, and dependencies strongly influence WS efficacy; noisy or sparse LFs create gaps between weakly supervised and gold-label performance.
• Procedural LF generators reveal that LF correlations and data-dependency materially affect the relative strengths of label models.
• Sequence tagging results show dependency-aware models (e.g., HMM/CHMM) often outperform MV, with some datasets favoring simpler approaches depending on coverage.