[Paper Review] Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing

Yuan Jin, Mark Carman+2 more|arXiv (Cornell University)|Feb 12, 2018

Mobile Crowdsensing and Crowdsourcing21 references3 citations

TL;DR

This paper proposes the SDR (Subjectivity-and-Difficulty Response) model, a probabilistic framework that explicitly models question difficulty and implicitly captures subjectivity through latent worker preference factors. By distinguishing subjective variation from objective difficulty, the model improves answer quality prediction and worker response prediction across diverse crowdsourcing datasets, while providing coherent, human-validated rankings of questions by difficulty and subjectivity.

ABSTRACT

The questions in a crowdsourcing task typically exhibit varying degrees of difficulty and subjectivity. Their joint effects give rise to the variation in responses to the same question by different crowd-workers. This variation is low when the question is easy to answer and objective, and high when it is difficult and subjective. Unfortunately, current quality control methods for crowdsourcing consider only the question difficulty to account for the variation. As a result,these methods cannot distinguish workers personal preferences for different correct answers of a partially subjective question from their ability/expertise to avoid objectively wrong answers for that question. To address this issue, we present a probabilistic model which (i) explicitly encodes question difficulty as a model parameter and (ii) implicitly encodes question subjectivity via latent preference factors for crowd-workers. We show that question subjectivity induces grouping of crowd-workers, revealed through clustering of their latent preferences. Moreover, we develop a quantitative measure of the subjectivity of a question. Experiments show that our model(1) improves the performance of both quality control for crowd-sourced answers and next answer prediction for crowd-workers,and (2) can potentially provide coherent rankings of questions in terms of their difficulty and subjectivity, so that task providers can refine their designs of the crowdsourcing tasks, e.g. by removing highly subjective questions or inappropriately difficult questions.

Motivation & Objective

• To address the limitation of existing quality control methods in crowdsourcing that conflate worker disagreement due to subjectivity with errors from difficulty.
• To model question subjectivity not as a fixed property but as latent worker preferences that cluster workers into groups based on response patterns.
• To develop a unified probabilistic framework that separately estimates difficulty and subjectivity, enabling better prediction of both correct answers and individual worker responses.
• To provide a quantitative measure of question subjectivity that aligns with human assessments.
• To support task design by enabling task providers to identify and refine highly subjective or overly difficult questions.

Proposed method

• The SDR model uses a probabilistic graphical model with latent variables to represent worker expertise and preference factors for each question.
• Question difficulty is modeled as a direct parameter influencing the probability of a correct response.
• Subjectivity is encoded implicitly through latent preference factors that cluster workers into groups based on their response patterns to partially subjective questions.
• The model employs a Bayesian inference approach to estimate worker expertise, preference factors, and question difficulty and subjectivity parameters simultaneously.
• It leverages answer correlation structures in multi-option questions to improve estimation accuracy and group detection.
• A novel coherence evaluation method compares model estimates with human-annotated rankings of difficulty and subjectivity, validating the model's interpretability.

Experimental results

Research questions

• RQ1 Can a probabilistic model effectively distinguish between worker disagreement caused by question subjectivity and that caused by question difficulty in crowdsourcing tasks?
• RQ2 How can subjectivity be modeled implicitly through latent worker preference factors, and does this lead to detectable clustering of workers?
• RQ3 To what extent does the SDR model improve the accuracy of predicting both correct answers and individual worker responses compared to existing baselines?
• RQ4 Can the model's estimates of difficulty and subjectivity be meaningfully correlated with human-annotated rankings of these properties?
• RQ5 Can the model support task design by identifying and ranking questions by their difficulty and subjectivity levels?

Key findings

• The SDR model significantly outperforms five baselines in predicting unseen worker responses, with a 1.5–2.5% improvement in average accuracy across seven partially subjective datasets.
• The Nemenyi post-hoc test confirmed that the performance difference between SDR and all other models (CDS, GLAD, DS, MdWC) was statistically significant at α=0.10.
• The model achieved a 0.7659 average accuracy in predicting unseen responses on the Fashion dataset, outperforming the next-best model (CDS) at 0.7621.
• There was a strong negative correlation (r ≈ -0.85) between SDR’s estimated subjectivity and human-assigned rankings, confirming its validity in measuring subjectivity.
• The model successfully identified three distinct worker groups in the fashion judgment task, corresponding to different preference patterns, revealing subjectivity-induced clustering.
• Human assessors’ rankings of difficulty and subjectivity strongly aligned with SDR’s estimates, with positive correlations of r ≈ 0.82 for difficulty and r ≈ 0.88 for subjectivity.