[Paper Review] Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

Hanlu Wu, Tengfei Ma+2 more|arXiv (Cornell University)|Oct 25, 2020

Mobile Crowdsensing and Crowdsourcing61 references4 citations

TL;DR

This paper proposes a heterogeneous graph neural network (HGNN) framework that models worker-task interactions in crowdsourcing by learning latent correlations between workers and tasks through a novel attention mechanism. By incorporating inter-worker and inter-task dependencies via a homogeneous attention layer, the model achieves state-of-the-art performance in label aggregation across 13 real-world datasets, significantly outperforming existing methods in accuracy and robustness.

ABSTRACT

Crowdsourcing has attracted much attention for its convenience to collect labels from non-expert workers instead of experts. However, due to the high level of noise from the non-experts, an aggregation model that learns the true label by incorporating the source credibility is required. In this paper, we propose a novel framework based on graph neural networks for aggregating crowd labels. We construct a heterogeneous graph between workers and tasks and derive a new graph neural network to learn the representations of nodes and the true labels. Besides, we exploit the unknown latent interaction between the same type of nodes (workers or tasks) by adding a homogeneous attention layer in the graph neural networks. Experimental results on 13 real-world datasets show superior performance over state-of-the-art models.

Motivation & Objective

• To address the challenge of noisy labels in crowdsourcing by improving label aggregation beyond simple majority voting.
• To model complex, multi-dimensional worker and task relationships that traditional methods fail to capture.
• To develop a representation learning framework that does not require explicit task features, enhancing generalization and scalability.
• To integrate latent correlations among workers and among tasks to improve inference of true labels.

Proposed method

• Construct a heterogeneous graph where workers and tasks are two distinct node types connected by edges indicating labeling assignments.
• Apply message-passing graph neural networks (MP-GNN) to aggregate neighborhood information across worker-task interactions.
• Introduce a novel homogeneous attention layer (COR layer) to model latent correlations among same-type nodes (workers or tasks), capturing inter-worker and inter-task dependencies.
• Use a multi-layer architecture (MP2 + COR + MP2) to iteratively refine node representations and predict true labels.
• Train the model end-to-end using cross-entropy loss on labeled data, with no need for ground-truth labels during inference.
• Fix the hidden dimension to 30 for all datasets to balance performance and generalization in final evaluations.

Experimental results

Research questions

• RQ1 Can a heterogeneous graph neural network effectively model worker-task interactions for improved label aggregation in crowdsourcing?
• RQ2 How do latent correlations among workers and among tasks affect the accuracy of true label inference?
• RQ3 Does incorporating inter-node dependencies through a homogeneous attention layer improve performance over standard message-passing GNNs?
• RQ4 How does the proposed method compare to state-of-the-art label aggregation models across diverse real-world datasets?
• RQ5 To what extent can the model generalize without requiring explicit task features or ground-truth labels?

Key findings

• The MP2+COR+MP2 model achieved the highest average accuracy across all 13 datasets, significantly outperforming baseline models.
• On the web dataset, the proposed model reached 97.34% accuracy, surpassing the next-best model by over 1.5 percentage points.
• The COR layer improved performance on 8 out of 13 datasets, demonstrating the value of modeling latent inter-worker and inter-task correlations.
• The model achieved strong performance even with only 5% training data, indicating high data efficiency.
• The ablation study confirmed that the COR layer provides unique benefits beyond 2-hop message passing, as it captures long-range dependencies.