[Paper Review] Optimal Algorithms for Crawling a Hidden Database in the Web

Cheng Sheng, Nan Zhang+2 more|arXiv (Cornell University)|Aug 1, 2012

Web Data Mining and Analysis4 references4 citations

TL;DR

This paper presents provably optimal algorithms for crawling hidden databases—web-accessible databases queried via dynamic interfaces rather than static links—by minimizing the number of queries needed to extract all tuples. The proposed methods, including rank-shrink for numeric data and lazy-slice-cover for categorical data, achieve asymptotic optimality and demonstrate strong empirical performance, with hybrid algorithms enabling progressive tuple retrieval.

ABSTRACT

A hidden database refers to a dataset that an organization makes accessible on the web by allowing users to issue queries through a search interface. In other words, data acquisition from such a source is not by following static hyper-links. Instead, data are obtained by querying the interface, and reading the result page dynamically generated. This, with other facts such as the interface may answer a query only partially, has prevented hidden databases from being crawled effectively by existing search engines. This paper remedies the problem by giving algorithms to extract all the tuples from a hidden database. Our algorithms are provably efficient, namely, they accomplish the task by performing only a small number of queries, even in the worst case. We also establish theoretical results indicating that these algorithms are asymptotically optimal -- i.e., it is impossible to improve their efficiency by more than a constant factor. The derivation of our upper and lower bound results reveals significant insight into the characteristics of the underlying problem. Extensive experiments confirm the proposed techniques work very well on all the real datasets examined.

Motivation & Objective

• Address the challenge of efficiently crawling hidden databases that are inaccessible via traditional link-based crawling due to interface-based query access and result limits.
• Develop algorithms that extract all tuples from a hidden database using the fewest possible queries, even in the worst-case scenario.
• Establish theoretical lower bounds to prove that the proposed algorithms are asymptotically optimal—no algorithm can improve query efficiency by more than a constant factor.
• Design practical, progressive crawling algorithms that deliver tuples incrementally, enabling early termination with useful results.
• Evaluate the performance of the algorithms across diverse real-world datasets, including Yahoo! Autos and NSF, under varying query result limits (k).

Proposed method

• Propose the rank-shrink algorithm for numeric-only datasets, recursively partitioning the data space using range queries to minimize redundant or overlapping results.
• Introduce slice-cover and its optimized variant, lazy-slice-cover, for categorical datasets, using hierarchical partitioning based on attribute values to cover all possible tuple combinations efficiently.
• Design a hybrid algorithm combining rank-shrink and lazy-slice-cover to handle mixed-attribute datasets (numeric and categorical), dynamically selecting the optimal strategy per query.
• Leverage theoretical analysis to derive upper and lower bounds on query complexity, proving asymptotic optimality of the algorithms under worst-case conditions.
• Implement heuristics such as lazy evaluation in slice-cover to reduce unnecessary queries and improve runtime performance without sacrificing correctness.
• Use progressive query execution to ensure that tuples are output incrementally, allowing the crawler to be interrupted early with a useful subset of results.

Experimental results

Research questions

• RQ1 What is the minimum number of queries required to extract all tuples from a hidden database with a fixed result limit k?
• RQ2 Can we design algorithms that are provably optimal in terms of query complexity, even in the worst-case input distribution?
• RQ3 How do attribute types (numeric, categorical, or mixed) affect the structure and efficiency of crawling algorithms?
• RQ4 Can we design a crawling algorithm that delivers results progressively, so that partial execution yields a meaningful subset of tuples?
• RQ5 How do heuristic optimizations like lazy evaluation in slice-cover improve practical performance without compromising theoretical guarantees?

Key findings

• The rank-shrink algorithm achieves asymptotic optimality for numeric datasets, with query complexity matching the theoretical lower bound up to a constant factor.
• The lazy-slice-cover algorithm outperforms both slice-cover and DFS in practice on categorical datasets like NSF, despite slice-cover being theoretically optimal in the worst case.
• The hybrid algorithm successfully combines rank-shrink and lazy-slice-cover, demonstrating robust performance across mixed-attribute datasets such as Yahoo! Autos and Adult.
• Experiments show that the hybrid algorithm exhibits linear progressiveness: at 20% of the total queries, it retrieves approximately 30% of the tuples, indicating consistent and predictable output over time.
• The query cost of rank-shrink grows nearly linearly with dimensionality d in practice, even though theory predicts a linear dependence, due to infrequent 3-way splits in real data (e.g., Adult-numeric).
• For datasets with high tuple redundancy (e.g., Yahoo! Autos with >64 identical tuples), no algorithm can fully extract the database when k is too small (e.g., k=64), highlighting the role of data distribution in query feasibility.