[Paper Review] CrowdGather: Entity Extraction over Structured Domains

Theodoros Rekatsinas, Amol Deshpande+1 more|arXiv (Cornell University)|Feb 24, 2015

Mobile Crowdsensing and Crowdsourcing36 references3 citations

TL;DR

CrowdGather proposes an adaptive, budget-aware framework for crowdsourced entity extraction over structured domains by leveraging hierarchical attribute queries to improve coverage and reduce cost. It uses statistical gain estimation and query overlap analysis to dynamically select high-value queries, achieving up to 4X higher entity yield than baselines under the same budget.

ABSTRACT

Crowdsourced entity extraction is often used to acquire data for many applications, including recommendation systems, construction of aggregated listings and directories, and knowledge base construction. Current solutions focus on entity extraction using a single query, e.g., only using "give me another restaurant", when assembling a list of all restaurants. Due to the cost of human labor, solutions that focus on a single query can be highly impractical. In this paper, we leverage the fact that entity extraction often focuses on {\em structured domains}, i.e., domains that are described by a collection of attributes, each potentially exhibiting hierarchical structure. Given such a domain, we enable a richer space of queries, e.g., "give me another Moroccan restaurant in Manhattan that does takeout". Naturally, enabling a richer space of queries comes with a host of issues, especially since many queries return empty answers. We develop new statistical tools that enable us to reason about the gain of issuing {\em additional queries} given little to no information, and show how we can exploit the overlaps across the results of queries for different points of the data domain to obtain accurate estimates of the gain. We cast the problem of {\em budgeted entity extraction} over large domains as an adaptive optimization problem that seeks to maximize the number of extracted entities, while minimizing the overall extraction costs. We evaluate our techniques with experiments on both synthetic and real-world datasets, demonstrating a yield of up to 4X over competing approaches for the same budget.

Motivation & Objective

• To address the high cost and poor coverage of single-query crowdsourced entity extraction in large, diverse domains.
• To enable adaptive querying over structured domains with hierarchical attributes to improve discovery of rare or less popular entities.
• To maximize the number of unique entities extracted under a fixed monetary budget.
• To develop statistical techniques that estimate the expected gain of issuing additional queries when little prior information is available.
• To exploit structural overlaps between query results to guide cost-effective query selection in sparse domains.

Proposed method

• Models structured domains as partially ordered sets (posets) where each node represents a query combining attribute values.
• Introduces a regression-based method to estimate the expected gain of issuing additional queries when initial results are sparse.
• Uses query result overlaps to infer potential gains and guide adaptive query selection across the poset.
• Applies an adaptive optimization algorithm that balances exploration (new queries) and exploitation (repeated high-gain queries) under budget constraints.
• Employs statistical tools to reason about the likelihood of discovering new entities without prior knowledge of the population size.
• Leverages known hierarchical structures in attributes (e.g., cuisine, location, takeout) to generate targeted, diverse queries.

Experimental results

Research questions

• RQ1 How can we efficiently explore large, structured data domains using crowdsourcing while minimizing cost?
• RQ2 What statistical techniques can estimate the expected gain of issuing additional queries when initial results are sparse?
• RQ3 How can we exploit overlaps between query results to guide adaptive query selection in sparse domains?
• RQ4 How does adaptive query selection over structured domains compare to fixed-query strategies in terms of entity yield and cost efficiency?
• RQ5 Can we achieve near-omniscient performance in entity extraction with limited budget and partial information?

Key findings

• CrowdGather achieves up to 4X higher entity yield compared to competing baseline approaches under the same budget.
• The framework reduces wasted queries on popular entities by dynamically shifting focus to less explored, high-potential query regions in the poset.
• In large, sparse domains, CrowdGather’s performance is within 25% of an omniscient strategy with perfect information.
• The regression-based gain estimation technique effectively predicts query utility even with minimal initial data.
• Exploiting structural overlaps between queries significantly improves the efficiency of query selection and reduces redundant labor.
• The adaptive strategy outperforms fixed-query and single-query approaches in both coverage and cost-effectiveness across synthetic and real-world datasets.