[Paper Review] Reducing Uncertainty of Schema Matching via Crowdsourcing with Accuracy Rates

Chen Zhang, Lei Chen+3 more|arXiv (Cornell University)|Sep 11, 2018

Mobile Crowdsensing and Crowdsourcing6 citations

TL;DR

This paper proposes two adaptive crowdsourcing frameworks—Single CCQ and Multiple CCQ—to reduce uncertainty in schema matching by dynamically selecting and publishing Correspondence Correctness Questions (CCQs) based on worker accuracy rates. It mathematically models uncertainty reduction as entropy gain and provides efficient algorithms with theoretical bounds, achieving significant uncertainty reduction within budget constraints.

ABSTRACT

Schema matching is a central challenge for data integration systems. Inspired by the popularity and the success of crowdsourcing platforms, we explore the use of crowdsourcing to reduce the uncertainty of schema matching. Since crowdsourcing platforms are most effective for simple questions, we assume that each Correspondence Correctness Question (CCQ) asks the crowd to decide whether a given correspondence should exist in the correct matching. Furthermore, members of a crowd may sometimes return incorrect answers with different probabilities. Accuracy rates of individual crowd workers are probabilities of returning correct answers which can be attributes of CCQs as well as evaluations of individual workers. We prove that uncertainty reduction equals to entropy of answers minus entropy of crowds and show how to obtain lower and upper bounds for it. We propose frameworks and efficient algorithms to dynamically manage the CCQs to maximize the uncertainty reduction within a limited budget of questions. We develop two novel approaches, namely `Single CCQ' and `Multiple CCQ', which adaptively select, publish and manage questions. We verify the value of our solutions with simulation and real implementation.

Motivation & Objective

• Address the inherent uncertainty in semi-automatic schema matching tools, which produce multiple possible matchings with associated probabilities.
• Overcome the limitations of relying on non-expert users for resolving ambiguity in ad hoc data integration scenarios.
• Leverage crowdsourcing platforms like Amazon Mechanical Turk to improve matching accuracy through human judgment on fine-grained correspondence correctness.
• Optimize the trade-off between cost (number of CCQs) and performance (uncertainty reduction) under a fixed budget.
• Model worker accuracy rates as probabilistic attributes to account for varying reliability of crowd contributors.

Proposed method

• Define Correspondence Correctness Questions (CCQs) as binary queries asking whether a specific attribute correspondence is correct.
• Model uncertainty reduction as the difference between the entropy of answers and the entropy of the crowd’s responses, proving this equals expected uncertainty reduction.
• Formulate the CCQ selection problem as maximizing expected uncertainty reduction, leveraging submodularity to design a (1+ε)-approximation algorithm.
• Introduce two frameworks: Single CCQ (selects one optimal CCQ per iteration) and Multiple CCQ (selects a batch of k CCQs per iteration) based on dynamic feedback.
• Incorporate worker accuracy rates as prior probabilities to weight the reliability of answers and refine uncertainty estimates iteratively.
• Derive theoretical lower and upper bounds for uncertainty reduction, enabling performance guarantees in the selection process.

Experimental results

Research questions

• RQ1 How can crowdsourcing be effectively used to reduce uncertainty in schema matching when automated tools fail to resolve ambiguities?
• RQ2 What is the optimal strategy for selecting CCQs to maximize uncertainty reduction while minimizing the number of questions posed?
• RQ3 How do worker accuracy rates influence the reliability of crowd-sourced answers and the overall uncertainty reduction process?
• RQ4 Can the uncertainty reduction be formally bounded, and what are the theoretical limits of improvement via crowdsourcing?
• RQ5 How can the selection of multiple CCQs at once be optimized under budget and time constraints, considering variable answer rates and worker reliability?

Key findings

• Uncertainty reduction is mathematically equivalent to the difference between the entropy of answers and the entropy of the crowd, providing a solid theoretical foundation.
• The CCQ selection problem is NP-hard, but a (1+ε)-approximation algorithm is proposed based on the submodular nature of the uncertainty reduction function.
• The Multiple CCQ framework outperforms Single CCQ in reducing uncertainty per question, especially when worker accuracy is modeled explicitly.
• Theoretical bounds for uncertainty reduction are derived, enabling performance evaluation and comparison across different strategies.
• Simulations and real-world implementation confirm that the proposed frameworks significantly reduce uncertainty with limited budget, outperforming baseline approaches.
• Worker accuracy rates are critical for reliable uncertainty estimation and must be modeled as part of the CCQ selection process to avoid bias from low-quality responses.