[论文解读] Consistent Weighted Sampling Made Fast, Small, and Easy

Bernhard Haeupler, Mark S. Manasse+1 more|arXiv (Cornell University)|Oct 16, 2014

Advanced Image and Video Retrieval Techniques参考文献 22被引用 27

一句话总结

本文提出了一种快速、紧凑且准确的加权Jaccard相似度估计方法，通过随机舍入将加权集合缩减为可调大小的无权集合。该方法仅需每个元素恒定数量的哈希计算，即可实现一次完成的近似独立样本计算，相较于先前方法实现高达两个数量级的速度提升，同时偏差可忽略不计，精度损失极小。

ABSTRACT

Document sketching using Jaccard similarity has been a workable effective technique in reducing near-duplicates in Web page and image search results, and has also proven useful in file system synchronization, compression and learning applications. Min-wise sampling can be used to derive an unbiased estimator for Jaccard similarity and taking a few hundred independent consistent samples leads to compact sketches which provide good estimates of pairwise-similarity. Subsequent works extended this technique to weighted sets and show how to produce samples with only a constant number of hash evaluations for any element, independent of its weight. Another improvement by Li et al. shows how to speedup sketch computations by computing many (near-)independent samples in one shot. Unfortunately this latter improvement works only for the unweighted case. In this paper we give a simple, fast and accurate procedure which reduces weighted sets to unweighted sets with small impact on the Jaccard similarity. This leads to compact sketches consisting of many (near-)independent weighted samples which can be computed with just a small constant number of hash function evaluations per weighted element. The size of the produced unweighted set is furthermore a tunable parameter which enables us to run the unweighted scheme of Li et al. in the regime where it is most efficient. Even when the sets involved are unweighted, our approach gives a simple solution to the densification problem that other works attempted to address. Unlike previously known schemes, ours does not result in an unbiased estimator. However, we prove that the bias introduced by our reduction is negligible and that the standard deviation is comparable to the unweighted case. We also empirically evaluate our scheme and show that it gives significant gains in computational efficiency, without any measurable loss in accuracy.

研究动机与目标

• 解决现有加权采样方法在元素权重增大时计算效率低下的问题。
• 实现加权集合的快速、单次完成近似独立样本计算，性能媲美无权方案。
• 通过随机舍入将加权集合缩减为无权集合，同时对Jaccard相似度的偏差可忽略不计。
• 引入可调参数以控制生成的无权集合大小，从而在下游草图处理中实现最优性能。
• 证明所提方法在高相似度情况下显著提升速度，且精度损失不可测量。

提出的方法

• 使用双尺度或更多尺度的随机舍入将加权集合缩减为无权集合，以实现高效采样。
• 应用Li等人[17]提出的一次置换技术，仅通过每个元素恒定数量的哈希计算，即可在单次遍历中计算出数百个近似独立样本。
• 使用可调参数控制生成的无权集合大小，以优化后续草图处理的效率。
• 引入一种阈值机制，当相似度低于用户定义的阈值α时跳过相似度估计，从而提升实际效率。
• 证明舍入引入的偏差可忽略不计，且估计误差的尾部界限与无权情况相当。
• 通过与Ioffe算法和随机舍入方法的对比，对方法进行实验评估，测量在不同相似度水平下的绝对误差和标准差。

实验结果

研究问题

• RQ1 加权Jaccard相似度估计能否在保持高精度的前提下实现显著加速？
• RQ2 将加权集合随机舍入为无权集合是否会在Jaccard相似度估计中引入显著偏差？
• RQ3 能否通过集合缩减将无权草图中的单次置换采样技术适配至加权集合？
• RQ4 缩减后的无权集合大小可调性如何影响计算效率和估计质量？
• RQ5 在高相似度场景下，计算速度与估计精度之间的权衡关系如何？

主要发现

• 所提方法相较于先前的加权采样方法，草图计算速度最高可提升两个数量级。
• 在高相似度值（例如0.96）下，该方法的平均绝对误差与Ioffe算法相当，误差低于0.01。
• 在Jaccard相似度介于0.8至0.9之间时，该方法在平均绝对误差方面略优于Ioffe算法。
• 在大多数相似度水平下，该方法的估计误差标准差小于或与Ioffe方法相当，表明其性能稳定。
• 随机舍入引入的偏差可忽略不计，且该方法保持了与无权情况相似的尾部界限。
• 即使在低相似度值（例如0.4）下，绝对误差仍低于0.035，对应平均额外错匹配约4个桶。