[Paper Review] Fault-Tolerant Entity Resolution with the Crowd

Anja Gruenheid, Besmira Nushi+3 more|arXiv (Cornell University)|Dec 2, 2015

Mobile Crowdsensing and Crowdsourcing38 references14 citations

TL;DR

This paper proposes a fault-tolerant framework for entity resolution (ER) using crowdsourced data, introducing a votes graph and MinMax decision function to interpret noisy worker inputs while preserving both positive and negative signals. It achieves at least 20% higher ER quality than consensus-based methods by optimizing task selection and parallelization under budget constraints.

ABSTRACT

In recent years, crowdsourcing is increasingly applied as a means to enhance data quality. Although the crowd generates insightful information especially for complex problems such as entity resolution (ER), the output quality of crowd workers is often noisy. That is, workers may unintentionally generate false or contradicting data even for simple tasks. The challenge that we address in this paper is how to minimize the cost for task requesters while maximizing ER result quality under the assumption of unreliable input from the crowd. For that purpose, we first establish how to deduce a consistent ER solution from noisy worker answers as part of the data interpretation problem. We then focus on the next-crowdsource problem which is to find the next task that maximizes the information gain of the ER result for the minimal additional cost. We compare our robust data interpretation strategies to alternative state-of-the-art approaches that do not incorporate the notion of fault-tolerance, i.e., the robustness to noise. In our experimental evaluation we show that our approaches yield a quality improvement of at least 20% for two real-world datasets. Furthermore, we examine task-to-worker assignment strategies as well as task parallelization techniques in terms of their cost and quality trade-offs in this paper. Based on both synthetic and crowdsourced datasets, we then draw conclusions on how to minimize cost while maintaining high quality ER results.

Motivation & Objective

• To address the challenge of unreliable, noisy crowd worker inputs in entity resolution (ER) while minimizing cost and maximizing result quality.
• To develop a robust data interpretation mechanism that handles conflicting and erroneous worker answers without discarding negative feedback.
• To solve the next-crowdsource problem by selecting optimal record pairs to query next, maximizing information gain per unit cost.
• To evaluate task assignment and parallelization strategies for cost-quality trade-offs in incremental, fault-tolerant ER.
• To demonstrate that fault-tolerant mechanisms outperform consensus-based approaches in real-world and synthetic datasets.

Proposed method

• Introduces a votes graph data structure to encode both positive and negative signals from crowd workers, capturing pairwise relationships and their confidence.
• Proposes the MinMax decision function as a fault-tolerant method to interpret noisy votes, ensuring consistency with worker signals even under high error rates.
• Adapts the MinMax matrix and votes graph incrementally as new crowd responses are collected, enabling online, incremental ER.
• Designs queuing strategies for the next-crowdsource problem that prioritize tasks based on expected information gain, minimizing uncertainty or error.
• Employs intra- and inter-task parallelization to improve efficiency and evaluate trade-offs between cost and result quality.
• Integrates the fault-tolerant interpretation and task selection into an incremental ER framework that maintains consistency and correctness over time.

Experimental results

Research questions

• RQ1 How can we interpret noisy, conflicting, and potentially malicious crowd worker responses in entity resolution without discarding negative feedback?
• RQ2 What is the optimal strategy for selecting the next record pair to crowdsource to maximize information gain while minimizing cost?
• RQ3 How do different task assignment and parallelization strategies affect the quality and cost of crowdsourced ER?
• RQ4 To what extent do fault-tolerant data interpretation mechanisms outperform consensus-based approaches in real-world ER scenarios?
• RQ5 Can fault-tolerant ER be generalized beyond crowdsourcing to other unreliable data sources?

Key findings

• The proposed MinMax decision function significantly improves ER quality by preserving and reasoning over both positive and negative worker signals, outperforming consensus-based methods by at least 20% on two real-world datasets.
• The next-crowdsource queuing strategies that prioritize uncertainty reduction or error minimization lead to better result quality with lower budget usage compared to random or heuristic task selection.
• Intra- and inter-task parallelization techniques improve execution efficiency without degrading result quality, enabling scalable deployment of the framework.
• The experimental evaluation on both synthetic and crowdsourced data (via Amazon Mechanical Turk) confirms that fault-tolerant mechanisms are robust under high worker error rates, up to 30%.
• The framework is generalizable beyond crowdsourcing and can be applied to any unreliable data source where data quality is uncertain.
• The study demonstrates that incorporating negative feedback and using fault-tolerant interpretation is critical—ignoring contradictions leads to suboptimal and inconsistent ER results.