[Paper Review] PASS-JOIN: A Partition-based Method for Similarity Joins

Guoliang Li, Dong Deng+2 more|arXiv (Cornell University)|Nov 30, 2011

Data Quality and Management10 references22 citations

TL;DR

PASS-JOIN is a partition-based method for efficient string similarity joins under edit-distance constraints, using segment-based substring selection and inverted indices to minimize candidate pairs. It outperforms state-of-the-art methods on both short and long strings by adaptively selecting substrings and applying novel pruning and verification techniques.

ABSTRACT

As an essential operation in data cleaning, the similarity join has attracted considerable attention from the database community. In this paper, we study string similarity joins with edit-distance constraints, which find similar string pairs from two large sets of strings whose edit distance is within a given threshold. Existing algorithms are efficient either for short strings or for long strings, and there is no algorithm that can efficiently and adaptively support both short strings and long strings. To address this problem, we propose a partition-based method called Pass-Join. Pass-Join partitions a string into a set of segments and creates inverted indices for the segments. Then for each string, Pass-Join selects some of its substrings and uses the selected substrings to find candidate pairs using the inverted indices. We devise efficient techniques to select the substrings and prove that our method can minimize the number of selected substrings. We develop novel pruning techniques to efficiently verify the candidate pairs. Experimental results show that our algorithms are efficient for both short strings and long strings, and outperform state-of-the-art methods on real datasets.

Motivation & Objective

• To address the lack of a unified, efficient method for string similarity joins across both short and long strings.
• To reduce the number of candidate pairs generated in similarity joins while ensuring no valid pairs are missed.
• To develop adaptive techniques that work efficiently for both short strings (e.g., names) and long strings (e.g., titles, abstracts).
• To improve verification efficiency through length-aware and prefix-sharing optimization techniques.

Proposed method

• Proposes a partition scheme that divides strings into segments, ensuring that similar strings share at least one segment.
• Uses inverted indices built on segments to efficiently locate candidate matching strings.
• Employs a position-aware and multi-match-aware substring selection strategy to minimize the number of selected substrings.
• Applies an extension-based verification method that reuses common prefix computations to speed up edit distance checks.
• Introduces pruning and early-termination techniques during verification to reduce redundant computation.
• Processes strings in sorted order, incrementally building indices and enabling efficient candidate lookup.

Experimental results

Research questions

• RQ1 Can a single method efficiently handle similarity joins for both short and long strings?
• RQ2 What is the minimal number of substrings needed to guarantee detection of all valid pairs under edit distance constraints?
• RQ3 How can candidate pairs be efficiently verified with minimal recomputation?
• RQ4 Can pruning and early termination significantly reduce verification cost in similarity joins?

Key findings

• PASS-JOIN achieves nearly linear scalability with dataset size, with 360s, 530s, and 700s elapsed times for 400K, 500K, and 600K strings respectively on the Author dataset (τ=4).
• On the Author dataset, PASS-JOIN uses only 1.92MB of index space, significantly less than ED-Join (25.34MB) and Trie-Join (16.32MB).
• PASS-JOIN outperforms ED-Join and Trie-Join on real datasets, demonstrating superior performance across both short and long string workloads.
• The multi-match-aware substring selection method minimizes the number of substrings required, proving theoretically optimal in candidate generation.
• The extension-based verification with shared prefix computation reduces verification time, especially for long strings.
• Experimental results confirm PASS-JOIN's adaptability and efficiency, making it suitable for systems requiring support across diverse string lengths.