[Paper Review] Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming

Cheng-Yu Hsieh, Jieyu Zhang+1 more|arXiv (Cornell University)|Mar 2, 2022

Educational Games and Gamification4 citations

TL;DR

Nemo proposes an interactive data programming framework that improves weak supervision by strategically selecting development data and contextualizing labeling functions (LFs) based on user interaction. By formalizing the LF development process as an iterative cycle, Nemo boosts productivity by 20% on average (up to 47%) compared to standard weak supervision pipelines.

ABSTRACT

Weak Supervision (WS) techniques allow users to efficiently create large training datasets by programmatically labeling data with heuristic sources of supervision. While the success of WS relies heavily on the provided labeling heuristics, the process of how these heuristics are created in practice has remained under-explored. In this work, we formalize the development process of labeling heuristics as an interactive procedure, built around the existing workflow where users draw ideas from a selected set of development data for designing the heuristic sources. With the formalism, we study two core problems of how to strategically select the development data to guide users in efficiently creating informative heuristics, and how to exploit the information within the development process to contextualize and better learn from the resultant heuristics. Building upon two novel methodologies that effectively tackle the respective problems considered, we present Nemo, an end-to-end interactive system that improves the overall productivity of WS learning pipeline by an average 20% (and up to 47% in one task) compared to the prevailing WS approach.

Motivation & Objective

• To formalize the labeling function (LF) development process in weak supervision as an interactive, data-driven workflow.
• To address the inefficiency of random development data selection in existing weak supervision pipelines.
• To model labeling functions not in isolation, but with contextual awareness of the data and user behavior during development.
• To improve the overall productivity and accuracy of weak supervision pipelines by integrating user interaction and data context.
• To bridge the gap between active learning and data programming by embedding user-guided development into the learning pipeline.

Proposed method

• Introduces Interactive Data Programming (IDP), a formal framework modeling LF development as an iterative cycle with two core components: strategic data selection and contextualized LF modeling.
• Proposes the Strategic Example Utility (SEU) metric to guide selection of development data points that maximize expected utility in LF creation.
• Employs a user model that estimates the probability of a user creating a new LF based on the presence of specific patterns (primitives) in the development data.
• Designs a contextualized learning pipeline that incorporates information from the development process—such as which data points were inspected and which patterns were used—into LF denoising and aggregation.
• Extends the SEU framework to support multi-example and multi-LF user interactions, enabling flexible interface design for complex LF authoring.
• Integrates a primitive-based example explorer in the Nemo UI, allowing users to inspect multiple examples containing a selected pattern to evaluate LF generalization potential.

Experimental results

Research questions

• RQ1 How can development data be strategically selected to maximize the informativeness and utility of labeling functions in weak supervision?
• RQ2 How can the context of the LF development process—such as user interactions and data patterns—be leveraged to improve the accuracy and generalization of labeling functions?
• RQ3 To what extent does integrating user-guided development into the weak supervision pipeline improve overall productivity and model performance?
• RQ4 Can a formalized, interactive workflow for LF development outperform traditional, passive weak supervision pipelines in terms of efficiency and effectiveness?
• RQ5 How can the system support users in creating multiple labeling functions per iteration while maintaining model accuracy and utility estimation?

Key findings

• Nemo improves the overall productivity of the weak supervision pipeline by an average of 20% across multiple datasets and tasks.
• In one specific task, Nemo achieves a 47% improvement in productivity compared to the prevailing weak supervision approach.
• The Strategic Example Utility (SEU) selection strategy significantly outperforms random sampling in guiding users toward more effective labeling function development.
• Contextualizing labeling functions using development process metadata leads to better denoising and more accurate probabilistic label aggregation.
• The integration of a primitive-based example explorer enhances user ability to evaluate LF generalization, reducing overfitting to idiosyncratic data patterns.
• Extending the framework to support multi-LF and multi-example development workflows maintains high utility and accuracy, demonstrating scalability and flexibility.