[Paper Review] Automatic Classification of Text Databases Through Query Probing

Panagiotis G. Ipeirotis, Luis Gravano+1 more|ArXiv.org|Jan 1, 2000

Information Retrieval and Search Behavior3 citations

TL;DR

This paper proposes an automated method to classify search-only text databases by generating probing queries from a rule-based document classifier, then classifying databases based on match counts. The approach achieves accurate, scalable classification without document retrieval, with experiments showing significant skew in results that confirms correct categorization at the 99.9% confidence level.

ABSTRACT

Many text databases on the web are 'hidden' behind search interfaces, and their documents are only accessible through querying. Search engines typically ignore the contents of such search-only databases. Recently, Yahoo-like directories have started to manually organize these databases into categories that users can browse to find these valuable resources. We propose a novel strategy to automate the classification of search-only text databases. Our technique starts by training a rule-based document classifier, and then uses the classifier's rules to generate probing queries. The queries are sent to the text databases, which are then classified based on the number of matches that they produce for each query. We report some initial exploratory experiments that show that our approach is promising to automatically characterize the contents of text databases accessible on the web.

Motivation & Objective

• To automate the classification of text databases hidden behind search interfaces, which are inaccessible to standard web crawlers.
• To eliminate manual categorization efforts by leveraging query probing as a scalable alternative to human-curated directories.
• To develop a method that classifies databases based on query match counts rather than document retrieval.
• To evaluate the effectiveness of query probing in accurately identifying the topical focus of web-based text databases.
• To explore hierarchical classification and improve query efficiency through classifier rule optimization.

Proposed method

• Train a rule-based document classifier (using RIPPER) on a labeled dataset to generate topic-specific rules.
• Transform each classifier rule into a query probe by extracting keywords and forming boolean queries.
• Submit these queries to the target text database and record the number of matching documents for each.
• Classify the database based on specificity (proportion of matches in a target category) and coverage (total matches across categories).
• Use a Chi-squared test to assess the statistical significance of the match distribution skew toward the correct category.
• Optimize query length by removing non-informative terms (e.g., terms returning zero results) to improve efficiency.

Experimental results

Research questions

• RQ1 Can query probing based on classifier rules accurately identify the topical focus of a search-only text database?
• RQ2 How effective is match-count analysis in classifying databases without retrieving documents?
• RQ3 What is the statistical significance of the observed skew in query match distributions across categories?
• RQ4 How does query length and rule complexity affect classification efficiency and accuracy?
• RQ5 Can this method scale to hierarchical classification and diverse search interface capabilities?

Key findings

• The method successfully classified four test databases (Cora, American Scientist, AllOutdoors, ReligionToday) into their correct categories using only query match counts.
• Specificity values clearly indicated that each database was focused on its intended category, with Cora showing 1450 matches for the 'Computers' category versus 151 for 'Science'.
• The Chi-squared test confirmed that the observed match distribution skew was statistically significant at the 99.9% confidence level for all databases.
• Simpler rule sets (29 rules, 32 words) outperformed more complex ones (31 rules, 92 words) in query efficiency, despite slightly lower accuracy.
• Truncating long queries by removing non-matching terms improved performance on systems with query length limits, such as the American Scientist database.
• The results demonstrate that a small set of well-chosen queries can effectively characterize the content of a search-only database without document retrieval.