[Paper Review] CrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designs

Jorge Ramírez, Simone Degiacomi+4 more|arXiv (Cornell University)|Sep 6, 2019

Mobile Crowdsensing and Crowdsourcing12 references4 citations

TL;DR

CrowdHub is a tool that extends crowdsourcing platforms like Figure Eight and Toloka to enable controlled, systematic evaluation of task designs by managing worker eligibility, population demographics, and execution timing. It reduces experimental bias—demonstrated by a 38% potential data loss in uncontrolled settings—through automated workflows, eligibility control, and time-sampling, improving data utility and reliability for researchers.

ABSTRACT

We present CrowdHub, a tool for running systematic evaluations of task designs on top of crowdsourcing platforms. The goal is to support the evaluation process, avoiding potential experimental biases that, according to our empirical studies, can amount to 38% loss in the utility of the collected dataset in uncontrolled settings. Using CrowdHub, researchers can map their experimental design and automate the complex process of managing task execution over time while controlling for returning workers and crowd demographics, thus reducing bias, increasing utility of collected data, and making more efficient use of a limited pool of subjects.

Motivation & Objective

• To address experimental biases in crowdsourcing task design evaluations caused by recurrent workers, condition crossover, timezones, and demographic imbalances.
• To support systematic, repeatable comparisons of multiple task designs using controlled worker populations and execution scheduling.
• To reduce data utility loss from uncontrolled evaluations, which can reach up to 38% due to bias from worker behavior and platform limitations.
• To provide researchers with a tool that automates the deployment and management of complex, multi-configuration task experiments on existing crowdsourcing platforms.
• To enable both between-subjects and within-subjects experimental designs with precise control over worker access and task sequencing.

Proposed method

• CrowdHub integrates with major crowdsourcing platforms (e.g., Figure Eight, Toloka) via their public APIs and JavaScript extensions for worker identification and metric collection.
• It uses a workflow editor to define task sequences, data flow, and experimental group assignments, supporting both sequential and parallel execution.
• Eligibility control policies are implemented to prevent condition crossover and manage returning workers, ensuring clean experimental groups.
• Population management allows requesters to assign quotas per worker subgroup (e.g., country, trust level) to avoid dominance by specific demographics.
• Time sampling schedules task execution across multiple time windows to reduce confounding effects from time-of-day or platform activity variations.
• The system parses experimental workflows and automatically deploys individual tasks with associated data units, instructions, and gold standards on the target platform.

Experimental results

Research questions

• RQ1 To what extent does worker recurrence introduce bias in task design evaluations, and how does it affect decision time and accuracy?
• RQ2 How does condition crossover—where workers experience multiple task conditions—impact performance and trust in task support features?
• RQ3 How do time-of-day and timezone variations affect worker performance consistency across repeated runs of the same task condition?
• RQ4 To what extent do demographic imbalances in worker populations (e.g., country dominance) compromise the validity of task design comparisons?
• RQ5 Can a controlled experimental framework reduce data utility loss compared to uncontrolled crowdsourcing evaluations?

Key findings

• Recurrent workers accounted for 38% of the dataset in an uncontrolled evaluation, showing lower decision times but no significant accuracy improvement, indicating potential learning effects without performance gains.
• Workers who crossed conditions—especially from poor to good highlighting support—showed increased decision times, likely due to reduced trust in the support, suggesting cognitive bias from prior exposure.
• Performance variability across runs of the same condition (e.g., decision time from 14s to 24s) demonstrated that time-of-day and platform activity confound comparisons in uncontrolled settings.
• Top contributing countries (Venezuela, Egypt, Ukraine) provided 48.1% of total judgments, indicating strong demographic imbalances that can skew results in uncontrolled experiments.
• The study found that uncontrolled evaluations can lead to up to a 38% loss in dataset utility due to experimental biases, highlighting the need for controlled frameworks.
• CrowdHub successfully mitigates these biases through eligibility control, population quotas, and time-sampling, enabling more reliable and efficient task design evaluation.