[Paper Review] GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling

Jing Li, Suiyi Ling+3 more|arXiv (Cornell University)|Mar 1, 2020

Mobile Crowdsensing and Crowdsourcing28 references4 citations

TL;DR

This paper proposes GPM, a generic probabilistic model that recovers ground truth labels and annotator behavior by modeling ground truth as a categorical distribution rather than a single value. It uses latent variables and EM-based MLE to detect unreliable annotators (e.g., random, repeated, or malicious) and outperforms state-of-the-art models in accuracy and robustness across discrete, continuous, and ordinal labeling tasks.

ABSTRACT

In the big data era, data labeling can be obtained through crowdsourcing. Nevertheless, the obtained labels are generally noisy, unreliable or even adversarial. In this paper, we propose a probabilistic graphical annotation model to infer the underlying ground truth and annotator's behavior. To accommodate both discrete and continuous application scenarios (e.g., classifying scenes vs. rating videos on a Likert scale), the underlying ground truth is considered following a distribution rather than a single value. In this way, the reliable but potentially divergent opinions from "good" annotators can be recovered. The proposed model is able to identify whether an annotator has worked diligently towards the task during the labeling procedure, which could be used for further selection of qualified annotators. Our model has been tested on both simulated data and real-world data, where it always shows superior performance than the other state-of-the-art models in terms of accuracy and robustness.

Motivation & Objective

• Address the challenge of noisy, unreliable, or adversarial labels in crowdsourced data labeling.
• Overcome limitations of existing models that assume a single ground truth value and fail to model diverse annotator behaviors.
• Enable robust ground truth inference across discrete, continuous, and ordinal labeling tasks by modeling ground truth as a categorical distribution.
• Detect and model different types of unreliable annotator behavior (random, repeated, malicious) while preserving valid divergent opinions from reliable annotators.
• Provide a generic, scalable framework applicable to real-world and simulated data with improved accuracy and robustness.

Proposed method

• Model ground truth as a categorical distribution instead of a single value to capture consensus and reliable divergent opinions.
• Introduce a latent variable to represent the probability that an annotator switches between reliable and unreliable behavior during labeling.
• Use Maximum Likelihood Estimation (MLE) via the Expectation-Maximization (EM) algorithm to jointly estimate ground truth distribution and annotator reliability.
• Model unreliable behaviors (random, repeated, inverted) using a uniform distribution over labels, enabling detection through parameter estimation.
• Estimate task difficulty via the entropy of the ground truth distribution, enabling active labeling strategies.
• Apply the model to both simulated and real-world datasets (e.g., Face emotion, MovieLens, Crowd Flower) for validation.

Experimental results

Research questions

• RQ1 Can a generic probabilistic model effectively recover ground truth labels across diverse labeling tasks (discrete, continuous, ordinal) by modeling ground truth as a distribution rather than a single value?
• RQ2 How well can the model detect and distinguish between different types of unreliable annotator behaviors (e.g., random, repeated, malicious) in real-world crowdsourcing?
• RQ3 To what extent does modeling reliable divergent opinions improve ground truth recovery compared to models that treat all disagreement as noise?
• RQ4 How does the model’s performance compare to state-of-the-art methods (e.g., D&S, GLAD, Ord-bin) in terms of accuracy and robustness on real-world datasets?
• RQ5 Can the estimated annotator reliability and task difficulty (via entropy) be used for downstream applications such as active labeling or annotator selection?

Key findings

• On the Face emotion dataset, the proposed model achieved a classification accuracy (F1) of 0.5431, outperforming D&S (0.5356) and GLAD (0.5155), and matching the best-performing Ord-bin model.
• On the MovieLens 20M dataset, the proposed model achieved a PLCC of 0.8620, SROCC of 0.8420, and RMSE of 0.2228, significantly outperforming D&S (PLCC: 0.4073, RMSE: 1.2287) and GLAD (RMSE: 0.6361).
• The model detected a spamminess ratio of 0% on the Crowd Dog dataset, indicating high data reliability, and outperformed D&S in robustness across multiple datasets.
• The model’s performance was consistently superior in continuous and ordinal labeling tasks, especially when compared to discrete-only models like D&S and GLAD.
• The estimated ground truth distribution captured reliable divergent opinions, and task difficulty was effectively inferred via entropy, enabling potential use in active learning.
• The model demonstrated robustness to irregular annotator behavior, with low spamminess ratios (1% on MovieLens, 0% on Crowd Dog), confirming its ability to identify and filter out low-quality contributions.