[Paper Review] Select Your Questions Wisely: For Entity Resolution With Crowd Errors

Vijaya Krishna Yalavarthi, Xiangyu Ke+1 more|arXiv (Cornell University)|Jan 28, 2017

Mobile Crowdsensing and Crowdsourcing50 references4 citations

TL;DR

This paper proposes PERC, a parameter-free framework for crowdsourced entity resolution that models worker errors using an uncertain graph with edge probabilities derived from crowd votes. By selecting the next crowdsourcing question based on global clustering reliability rather than local features, PERC improves entity resolution accuracy by 15% and reduces crowdsourcing cost by 50% compared to state-of-the-art methods.

ABSTRACT

Crowdsourcing is becoming increasingly important in entity resolution tasks due to their inherent complexity such as clustering of images and natural language processing. Humans can provide more insightful information for these difficult problems compared to machine-based automatic techniques. Nevertheless, human workers can make mistakes due to lack of domain expertise or seriousness, ambiguity, or even due to malicious intents. The state-of-the-art literature usually deals with human errors via majority voting or by assigning a universal error rate over crowd workers. However, such approaches are incomplete, and often inconsistent, because the expertise of crowd workers are diverse with possible biases, thereby making it largely inappropriate to assume a universal error rate for all workers over all crowdsourcing tasks. To this end, we mitigate the above challenges by considering an uncertain graph model, where the edge probability between two records A and B denotes the ratio of crowd workers who voted Yes on the question if A and B are same entity. In order to reflect independence across different crowdsourcing tasks, we apply the well-established notion of possible worlds, and develop parameter-free algorithms both for next crowdsourcing, as well as for entity resolution problems. In particular, using our framework, the problem of entity resolution becomes equivalent to finding the maximum-likelihood clustering; whereas for the next crowdsourcing, we identify the record pair that maximally increases the reliability of the maximum-likelihood clustering. Based on detailed empirical analysis over real-world datasets, we find that our proposed solution, PERC (probabilistic entity resolution with imperfect crowd) improves the quality by 15% and reduces the overall cost by 50% for the crowdsourcing-based entity resolution problem.

Motivation & Objective

• Address the limitations of existing crowdsourced entity resolution methods that assume universal error rates or use majority voting, which fail to account for worker expertise diversity and biases.
• Develop an end-to-end pipeline that jointly models entity resolution and crowd errors, avoiding treating quality assurance as an orthogonal problem.
• Minimize crowdsourcing cost while maximizing entity resolution accuracy by intelligently selecting the next record pair to query based on global clustering reliability.
• Eliminate the need for user-defined thresholds or prior knowledge of worker error rates, enabling practical deployment in real-world settings.

Proposed method

• Model entity resolution as an uncertain graph where edge probabilities represent the fraction of workers who answered 'YES' to whether two records refer to the same entity.
• Use possible worlds semantics to represent all potential consistent labelings of the uncertain graph, enabling probabilistic reasoning over worker responses.
• Define a reliability metric that captures the connectedness within and across clusters, serving as the objective for selecting the next optimal crowdsourcing question.
• Formulate the next question selection as maximizing the expected improvement in clustering reliability, leveraging edge independence to scale efficiently.
• Design a parameter-free algorithm that dynamically selects the most informative record pair to query next, based on its impact on overall clustering reliability.
• Integrate the reliability-based selection with correlation clustering for entity resolution, ensuring robustness to noisy or inconsistent worker inputs.

Experimental results

Research questions

• RQ1 How can we effectively model and mitigate the impact of heterogeneous worker errors in crowdsourced entity resolution without assuming a universal error rate?
• RQ2 What criteria should guide the selection of the next crowdsourcing question to maximize entity resolution accuracy while minimizing cost?
• RQ3 Can a global, reliability-based metric outperform local, heuristic-based metrics (e.g., path length or edge count) in guiding next-question selection?
• RQ4 To what extent can a parameter-free approach reduce dependency on domain-specific thresholds or prior error estimates in real-world entity resolution tasks?
• RQ5 How does the proposed method compare in accuracy and cost efficiency to state-of-the-art methods like MinMax, PC-Pivot, and DENSE under varying batch sizes and datasets?

Key findings

• PERC improves entity resolution accuracy by 15% compared to the state-of-the-art MinMax, DENSE, and PC-Pivot methods across four real-world datasets.
• PERC reduces the number of required crowdsourced questions by 50% on average, significantly lowering the overall cost of entity resolution.
• On the Gymnastics dataset, PERC reduces crowdsourcing cost by 55.56% compared to PC-Pivot and 38.46% compared to MinMax at F1-measure = 0.9.
• PERC achieves higher accuracy with fewer questions, demonstrating that reliability-based question selection is more effective than local feature-based approaches.
• Smaller batch sizes improve accuracy and reduce cost, and PERC consistently outperforms bDENSE across all tested batch sizes.
• The next-question selection time for PERC is an order of magnitude faster than state-of-the-art methods, indicating high computational efficiency.