[Paper Review] Optimization in Knowledge-Intensive Crowdsourcing

Senjuti Basu Roy, Ioanna Lykourentzou+3 more|arXiv (Cornell University)|Jan 7, 2014

Mobile Crowdsensing and Crowdsourcing27 references17 citations

TL;DR

SmartCrowd proposes a framework for optimizing worker-to-task assignment in knowledge-intensive crowdsourcing by modeling human factors such as expertise, wage expectations, and task acceptance probability. It uses pre-indexed, adaptively maintained C-dex structures to enable efficient, high-quality task assignment with theoretical guarantees and validated performance in real and synthetic experiments.

ABSTRACT

We present SmartCrowd, a framework for optimizing collaborative knowledge-intensive crowdsourcing. SmartCrowd distinguishes itself by accounting for human factors in the process of assigning tasks to workers. Human factors designate workers' expertise in different skills, their expected minimum wage, and their availability. In SmartCrowd, we formulate task assignment as an optimization problem, and rely on pre-indexing workers and maintaining the indexes adaptively, in such a way that the task assignment process gets optimized both qualitatively, and computation time-wise. We present rigorous theoretical analyses of the optimization problem and propose optimal and approximation algorithms. We finally perform extensive performance and quality experiments using real and synthetic data to demonstrate that adaptive indexing in SmartCrowd is necessary to achieve efficient high quality task assignment.

Motivation & Objective

• Address the lack of optimization in knowledge-intensive crowdsourcing (KI-C), where collaborative knowledge creation requires high-quality, cost-effective worker-task assignments.
• Formalize the worker-to-task assignment problem as an optimization task considering multiple skills, cost constraints, and human factors like expertise, wage, and acceptance probability.
• Design a pre-indexing mechanism (C-dex) to enable efficient real-time assignment while maintaining quality and cost objectives.
• Ensure adaptability to dynamic environments where workers join, leave, or change profiles, without preemption of assigned workers.
• Provide both optimal and approximation algorithms with theoretical guarantees, including NP-completeness, submodularity, and monotonicity under specific conditions.

Proposed method

• Formulate the KI-C assignment as a constrained optimization problem with objectives to meet quality thresholds, stay within cost limits, and avoid worker over/under-utilization.
• Introduce C-dex, a pre-computed index structure that organizes workers by skill profiles and wage expectations to accelerate assignment decisions.
• Use probabilistic modeling to incorporate worker acceptance ratios into the optimization, reflecting real-world uncertainty in task assignment.
• Propose an optimal solution via integer linear programming (ILP) for C-dex construction and maintenance under static conditions.
• Design two approximation strategies: a greedy algorithm with polynomial time complexity and constant-factor approximation, and C-dex+, which clusters similar workers into 'virtual workers' to improve efficiency.
• Implement adaptive index maintenance to handle dynamic changes in worker availability, profiles, and task arrivals without preemption.

Experimental results

Research questions

• RQ1 How can worker-to-task assignment in knowledge-intensive crowdsourcing be formally modeled as an optimization problem that accounts for multiple human factors?
• RQ2 What is the computational complexity of the C-dex index design problem, and can it be solved efficiently with theoretical guarantees?
• RQ3 How effective is pre-indexing (C-dex) in improving the efficiency and quality of task assignment compared to real-time optimization?
• RQ4 What are the performance and quality trade-offs of approximation algorithms like the greedy method and C-dex+ under dynamic worker and task conditions?
• RQ5 To what extent does adaptive index maintenance maintain assignment quality and efficiency in volatile, real-world crowdsourcing environments?

Key findings

• The C-dex index design problem is NP-complete, as proven by reduction from the Multiple Knapsack Problem, establishing the computational hardness of the core optimization task.
• The C-dex structure enables significant speedups in task assignment by pre-computing worker-task compatibility, reducing real-time computation time while maintaining high assignment quality.
• The greedy approximation algorithm for C-dex achieves a constant-factor approximation under certain conditions, ensuring theoretical performance bounds despite being suboptimal.
• C-dex+ outperforms standard C-dex by leveraging clustering of similar workers into virtual workers, reducing index size and improving scalability with minimal quality loss.
• Extensive experiments with real workers on a modified Amazon Mechanical Turk setup show that SmartCrowd maintains high task quality even under dynamic conditions, with performance stable across varying workloads and acceptance ratios.
• Simulation results confirm that adaptive index maintenance is essential—static indexing leads to degraded performance and quality in dynamic environments with changing worker availability and profiles.