[Paper Review] Comprehensive and Efficient Data Labeling via Adaptive Model Scheduling

Mu Yuan, Lan Zhang+2 more|arXiv (Cornell University)|Feb 8, 2020

Advanced Neural Network Applications36 references4 citations

TL;DR

This paper proposes an adaptive model scheduling framework that uses deep reinforcement learning to predict the value of unexecuted models and dynamically schedules model execution to maximize high-confidence label extraction under resource and delay constraints. Evaluated on five image datasets and 30 models, it achieves 53% faster processing with no loss in label quality, significantly improving efficiency in comprehensive data labeling workflows.

ABSTRACT

Labeling data (e.g., labeling the people, objects, actions and scene in images) comprehensively and efficiently is a widely needed but challenging task. Numerous models were proposed to label various data and many approaches were designed to enhance the ability of deep learning models or accelerate them. Unfortunately, a single machine-learning model is not powerful enough to extract various semantic information from data. Given certain applications, such as image retrieval platforms and photo album management apps, it is often required to execute a collection of models to obtain sufficient labels. With limited computing resources and stringent delay, given a data stream and a collection of applicable resource-hungry deep-learning models, we design a novel approach to adaptively schedule a subset of these models to execute on each data item, aiming to maximize the value of the model output (e.g., the number of high-confidence labels). Achieving this lofty goal is nontrivial since a model's output on any data item is content-dependent and unknown until we execute it. To tackle this, we propose an Adaptive Model Scheduling framework, consisting of 1) a deep reinforcement learning-based approach to predict the value of unexecuted models by mining semantic relationship among diverse models, and 2) two heuristic algorithms to adaptively schedule the model execution order under a deadline or deadline-memory constraints respectively. The proposed framework doesn't require any prior knowledge of the data, which works as a powerful complement to existing model optimization technologies. We conduct extensive evaluations on five diverse image datasets and 30 popular image labeling models to demonstrate the effectiveness of our design: our design could save around 53\% execution time without loss of any valuable labels.

Motivation & Objective

• To address the challenge of efficiently and comprehensively labeling large-scale data using multiple resource-intensive deep learning models.
• To minimize computation waste and execution time while maximizing the number of high-confidence labels extracted from each data item.
• To develop a framework that operates without prior knowledge of data content, adapting model scheduling in real time to content-dependent model outputs.
• To provide a lightweight, orthogonal solution that complements existing model optimization and acceleration techniques.
• To enable practical deployment in resource-constrained environments such as mobile photo apps and real-time data processing pipelines.

Proposed method

• A deep reinforcement learning (DRL) agent is trained to predict the value of unexecuted models by mining semantic relationships across diverse models through a learned model-relationship graph.
• The framework constructs a model-relationship graph to encode inter-model dependencies and similarities, enabling informed predictions about which models will yield high-value outputs.
• Two heuristic scheduling algorithms are proposed: one for deadline-constrained scenarios and another for deadline-memory constrained environments, optimizing model execution order based on predicted value.
• The DRL agent operates with minimal overhead—only 3–6ms inference time and 100MB CPU memory—making it practical for real-time deployment.
• Model execution is dynamically scheduled per data item, avoiding unnecessary runs by predicting which models will produce valuable, high-confidence labels.
• The system operates without requiring prior data distribution knowledge, adapting to content-dependent model behavior in real time.

Experimental results

Research questions

• RQ1 How can we predict the value of unexecuted models in a content-dependent, data-agnostic environment to avoid unnecessary computation?
• RQ2 What scheduling strategy maximizes the number of high-confidence labels extracted under strict time and memory constraints?
• RQ3 How can we effectively model and exploit semantic relationships among diverse deep learning models to improve prediction accuracy of model utility?
• RQ4 Can a lightweight DRL-based framework achieve significant efficiency gains without sacrificing label completeness in real-world data labeling workloads?
• RQ5 What is the trade-off between scheduling intelligence and computational overhead in large-scale model execution pipelines?

Key findings

• The proposed framework reduces model execution time by approximately 53% on average across five diverse image datasets without losing any valuable labels.
• The DRL agent achieves high prediction accuracy for unexecuted model value with only 3–6ms inference time per decision, making it suitable for real-time deployment.
• The model-relationship graph enables effective transfer of knowledge across models, improving prediction reliability even with limited training data.
• The heuristic scheduling algorithms significantly outperform baseline strategies such as random or fixed-order execution in terms of label quality and efficiency.
• The framework is computationally lightweight, consuming only 100MB CPU memory, compared to 500MB–8GB GPU memory per deep learning model, ensuring low overhead.
• The approach is orthogonal to existing model optimization techniques and can be seamlessly integrated into existing data labeling pipelines to enhance performance.