[Paper Review] On the Impact of Random Index-Partitioning on Index Compression

Moran Feldman, Ronny Lempel+2 more|arXiv (Cornell University)|Jul 28, 2011

Algorithms and Data Compression20 references3 citations

TL;DR

This paper investigates how random index-partitioning across servers degrades the effectiveness of document reordering techniques in inverted index compression. It shows that while such partitioning reduces index size logarithmically under random ordering, it diminishes the compression gains of advanced ordering schemes like lexical URL sorting and TSP-based clustering, ultimately favoring random distribution due to significant improvements in distributed query processing time despite worse compression.

ABSTRACT

The performance of processing search queries depends heavily on the stored index size. Accordingly, considerable research efforts have been devoted to the development of efficient compression techniques for inverted indexes. Roughly, index compression relies on two factors: the ordering of the indexed documents, which strives to position similar documents in proximity, and the encoding of the inverted lists that result from the ordered stream of documents. Large commercial search engines index tens of billions of pages of the ever growing Web. The sheer size of their indexes dictates the distribution of documents among thousands of servers in a scheme called local index-partitioning, such that each server indexes only several millions pages. Due to engineering and runtime performance considerations, random distribution of documents to servers is common. However, random index-partitioning among many servers adversely impacts the resulting index sizes, as it decreases the effectiveness of document ordering schemes. We study the impact of random index-partitioning on document ordering schemes. We show that index-partitioning decreases the aggregated size of the inverted lists logarithmically with the number of servers, when documents within each server are randomly reordered. On the other hand, the aggregated partitioned index size increases logarithmically with the number of servers, when state-of-the-art document ordering schemes, such as lexical URL sorting and clustering with TSP, are applied. Finally, we justify the common practice of randomly distributing documents to servers, as we qualitatively show that despite its ill-effects on the ensuing compression, it decreases key factors in distributed query evaluation time by an order of magnitude as compared with partitioning techniques that compress better.

Motivation & Objective

• To evaluate the impact of random index-partitioning on the compression effectiveness of document reordering techniques in inverted indexes.
• To quantify the trade-off between index size reduction and query processing time in distributed search systems.
• To assess whether state-of-the-art document ordering heuristics (e.g., lexical URL sorting, TSP-based clustering) remain effective under random partitioning across thousands of servers.
• To justify the industry practice of random document distribution despite its negative impact on compression by demonstrating its benefits in reducing query evaluation time.

Proposed method

• The study uses the 25M-page TREC .gov2 corpus to simulate index-partitioning across varying numbers of servers (m), with documents randomly assigned to servers.
• It evaluates multiple document reordering strategies: random assignment, lexical URL sorting, and TSP-based clustering, measuring the resulting compressed inverted list sizes.
• The paper applies dGap encoding (e.g., Delta encoding) to compress postings lists and measures total index size as a function of the number of servers.
• It models query processing time using a balls-into-urns framework, where postings are treated as balls and servers as urns, and derives probabilistic bounds on maximum load per server.
• Chernoff bounds are used to analyze the expected maximum load (i.e., query processing time) in disjunctive and conjunctive queries under random partitioning.
• The analysis compares theoretical bounds with empirical measurements on real TREC query workloads to validate the model.

Experimental results

Research questions

• RQ1 How does random index-partitioning affect the compressed size of inverted indexes when using advanced document reordering techniques?
• RQ2 To what extent does random partitioning diminish the compression advantage of lexical URL sorting and TSP-based clustering?
• RQ3 What is the trade-off between index size and query processing time under random versus URL-based document partitioning?
• RQ4 How does the number of servers influence the performance gap between effective and random document ordering in partitioned indexes?
• RQ5 Can the observed benefits of random partitioning in reducing query processing time outweigh its negative impact on index compression efficiency?

Key findings

• Random index-partitioning causes the aggregated size of compressed inverted lists to decrease logarithmically with the number of servers when document IDs are randomly assigned.
• In contrast, when state-of-the-art ordering schemes (e.g., lexical URL sorting, TSP clustering) are applied, the aggregated index size increases logarithmically with the number of servers due to reduced locality in partitioned collections.
• Despite reducing compression efficiency, random document partitioning decreases the maximum load per server (a proxy for query processing time) by an order of magnitude compared to URL-based partitioning schemes.
• The performance gain in query processing time under random partitioning is primarily due to better load balancing across servers, as shown by the balls-into-urns model and empirical measurements.
• The study finds that clustering documents by host (e.g., same web host) contributes most to compression gains, while further lexical URL sorting within hosts yields only marginal improvements.
• The results suggest that global term-partitioning (i.e., partitioning by terms rather than documents) would outperform current random document-partitioning in terms of index size, but at the cost of increased query coordination overhead.