[Paper Review] Even Better Framework for min-wise Based Algorithms

Guy Feigenblat, Ely Porat+1 more|arXiv (Cornell University)|Feb 17, 2011

Advanced Image and Video Retrieval Techniques23 references3 citations

TL;DR

This paper introduces a novel d-k-min-wise independent family of hash functions that achieves exponential time and space improvements in min-wise based algorithms using only O(d) independence, significantly reducing the required independence degree compared to prior work. The key contribution is an efficient construction that requires just 8-wise independence for most practical cases, bypassing the O(log(1/ε)) lower bound and enabling optimal, constant-degree independence for error-bounded approximations.

ABSTRACT

In a recent paper from SODA11 \cite{kminwise} the authors introduced a general framework for exponential time improvement of \minwise based algorithms by defining and constructing almost \kmin independent family of hash functions. Here we take it a step forward and reduce the space and the independent needed for representing the functions, by defining and constructing a \dkmin independent family of hash functions. Surprisingly, for most cases only 8-wise independent is needed for exponential time and space improvement. Moreover, we bypass the $O(\log{\frac{1}ε})$ independent lower bound for approximately \minwise functions \cite{patrascu10kwise-lb}, as we use alternative definition. In addition, as the independent's degree is a small constant it can be implemented efficiently. Informally, under this definition, all subsets of size $d$ of any fixed set $X$ have an equal probability to have hash values among the minimal $k$ values in $X$, where the probability is over the random choice of hash function from the family. This property measures the randomness of the family, as choosing a truly random function, obviously, satisfies the definition for $d=k=|X|$. We define and give an efficient time and space construction of approximately \dkmin independent family of hash functions. The degree of independent required is optimal, i.e. only $O(d)$ for $2 \le d < k=O(\frac{d}{ε^2})$, where $ε\in (0,1)$ is the desired error bound. This construction can be used to improve many \minwise based algorithms, such as \cite{sizeEstimationFramework,Datar02estimatingrarity,NearDuplicate,SimilaritySearch,DBLP:conf/podc/CohenK07}, as will be discussed here. To our knowledge such definitions, for hash functions, were never studied and no construction was given before.

Motivation & Objective

• To address the high space and independence requirements of existing min-wise independent hash functions in algorithms like similarity estimation and duplicate detection.
• To reduce the independence degree needed for approximately min-wise hashing from O(log(1/ε)) to O(d), where d is the subset size.
• To define and construct a d-k-min-wise independent family that ensures all d-sized subsets of any set X have equal probability to achieve the k smallest hash values.
• To provide an efficient, space- and time-optimal construction of approximately d-k-min-wise independent hash functions for practical use in data mining and sketching.

Proposed method

• Introduce a new definition of d-k-min-wise independence: for any set X and subset Y ⊆ X of size d, the probability that h(y) is among the k smallest values in h(X) is exactly 1 / (|X| choose d), up to a (1±ε) error.
• Construct an approximately d-k-min-wise independent family using l-wise independent hash functions with l ≥ 3d+2, where l is a small constant.
• Prove that l-wise independence suffices for (ε / c)-d-k-min-wise independence, with c a universal constant, by analyzing rank distributions and block probabilities.
• Use telescoping sum techniques and bounds on tail probabilities of rank blocks to control error terms in the deviation from uniformity.
• Leverage the fact that only O(d) independence is needed for d-k-min-wise independence, which is optimal and significantly improves upon prior O(log(1/ε)) bounds.
• Apply the construction to improve existing min-wise based algorithms such as similarity estimation, sketching, and duplicate detection, with exponential gains in efficiency.

Experimental results

Research questions

• RQ1 Can the independence degree required for approximately min-wise hashing be reduced below O(log(1/ε)) while maintaining (1±ε) accuracy?
• RQ2 Is it possible to define and construct a d-k-min-wise independent family of hash functions that ensures uniform probability for all d-sized subsets to be among the k smallest hash values?
• RQ3 What is the minimal independence degree required to achieve (ε / c)-d-k-min-wise independence, and can it be made optimal and constant for practical d?
• RQ4 How can the construction be made space- and time-efficient while maintaining strong probabilistic guarantees for min-wise hashing applications?
• RQ5 Can this framework be applied to improve the performance of existing min-wise based algorithms like similarity estimation and sketching with exponential speedups?

Key findings

• The paper constructs an approximately d-k-min-wise independent family of hash functions using only O(d) independence, which is optimal and a significant improvement over the O(log(1/ε)) bound previously considered necessary.
• For most practical cases, only 8-wise independence is required to achieve exponential time and space improvements in min-wise based algorithms.
• The construction bypasses the O(log(1/ε)) independence lower bound from prior work by introducing a new definition of d-k-min-wise independence that does not rely on the same constraints.
• The method achieves (ε / c)-d-k-min-wise independence using l-wise independent hash functions with l ≥ 3d+2, where c is a universal constant.
• The theoretical analysis proves that the error deviation is bounded by ε · (k choose d) / (n choose d), ensuring high accuracy for subset sampling and similarity estimation.
• The framework enables exponential improvements in time and space complexity for a wide range of min-wise based algorithms, including similarity estimation, sketching, and duplicate detection.