[Paper Review] Digging Deeper into Deep Web Databases by Breaking Through the Top-k Barrier

Saravanan Thirumuruganathan, Nan Zhang+1 more|arXiv (Cornell University)|Aug 19, 2012

Data Management and Algorithms20 references4 citations

TL;DR

This paper proposes GetNext, a meta-algorithm that retrieves the (h+1)th top-ranked tuple from a hidden web database using only its public top-k interface, without prior knowledge of the ranking function. By iteratively generating reformulated queries and testing candidates through result analysis, GetNext enables access to more than k tuples—breaking the top-k barrier—demonstrated to retrieve 250 Amazon DVD titles in under 500 queries, well below Amazon's rate limit.

ABSTRACT

A large number of web databases are only accessible through proprietary form-like interfaces which require users to query the system by entering desired values for a few attributes. A key restriction enforced by such an interface is the top-k output constraint - i.e., when there are a large number of matching tuples, only a few (top-k) of them are preferentially selected and returned by the website, often according to a proprietary ranking function. Since most web database owners set k to be a small value, the top-k output constraint prevents many interesting third-party (e.g., mashup) services from being developed over real-world web databases. In this paper we consider the novel problem of "digging deeper" into such web databases. Our main contribution is the meta-algorithm GetNext that can retrieve the next ranked tuple from the hidden web database using only the restrictive interface of a web database without any prior knowledge of its ranking function. This algorithm can then be called iteratively to retrieve as many top ranked tuples as necessary. We develop principled and efficient algorithms that are based on generating and executing multiple reformulated queries and inferring the next ranked tuple from their returned results. We provide theoretical analysis of our algorithms, as well as extensive experimental results over synthetic and real-world databases that illustrate the effectiveness of our techniques.

Motivation & Objective

• To address the limitation of top-k web database interfaces that restrict access to only a few ranked results, hindering third-party applications like mashups and data analytics.
• To enable retrieval of more than k top-ranked tuples from hidden web databases without requiring private access or cooperation from database owners.
• To develop an automated, scalable solution that works solely through public query interfaces, avoiding costly and rate-limited crawling.
• To overcome the challenge of unknown ranking functions in hidden databases by inferring tuple order through query result analysis.

Proposed method

• Proposes a two-step process: candidate generation and candidate testing, to identify the next highest-ranked tuple beyond the top-k results.
• Generates multiple reformulated queries by varying attribute values to probe the database interface and collect result sets.
• Uses a candidate testing phase to determine whether a tuple is likely to be ranked higher than others not yet retrieved, based on result set overlaps and inclusion patterns.
• Employs a partial order inference mechanism to handle cases where full ordering is impossible due to query selectivity.
• Utilizes principles from infrequent-minimal-itemset mining to efficiently identify candidate tuples that could be ranked beyond the top-k.
• Implements ORDERED-GETNEXT and BEYOND-h-GETNEXT algorithms to optimize query cost and retrieval efficiency.

Experimental results

Research questions

• RQ1 Can we retrieve tuples ranked beyond the top-k limit of a hidden web database using only its public interface and without knowledge of its ranking function?
• RQ2 What is the minimal number of queries required to reliably identify the (h+1)th ranked tuple when the ranking function is unknown?
• RQ3 How can we design a systematic method to generate and test candidate tuples for inclusion in the extended top-h result set?
• RQ4 To what extent does query selectivity affect the performance and completeness of the retrieval process?
• RQ5 Can the proposed method scale to real-world databases with rate limits and limited query access?

Key findings

• The GetNext algorithm successfully retrieves the top-250 Amazon DVD titles—beyond Amazon’s default top-100—using fewer than 500 queries, well within Amazon’s 2000 queries per hour per IP limit.
• The algorithm achieves high accuracy in inferring the true ranking order, with Kendall-τ correlation increasing as query selectivity decreases, indicating better performance on less selective queries.
• The candidate testing phase effectively reduces false positives by leveraging result set overlaps, significantly improving retrieval precision.
• The method scales efficiently: as query selectivity decreases, the number of required queries drops due to earlier termination in the candidate testing phase.
• ORDERED-GETNEXT outperforms BEYOND-h-GETNEXT in query efficiency, especially for larger values of h and k, demonstrating its robustness across diverse workloads.