[Paper Review] An Ensemble Blocking Scheme for Entity Resolution of Large and Sparse Datasets

Janani Balaji, Faizan Javed+4 more|arXiv (Cornell University)|Sep 20, 2016

Data Quality and Management6 references3 citations

TL;DR

This paper proposes an ensemble blocking scheme that combines multiple blocking techniques to improve entity resolution in large, sparse datasets. By integrating locality-sensitive hashing (LSH) and rule-based blocking, the method enhances coverage and precision, reducing false negatives while maintaining scalability on massive, heterogeneous data sources like those at CareerBuilder.

ABSTRACT

Entity Resolution, also called record linkage or deduplication, refers to the process of identifying and merging duplicate versions of the same entity into a unified representation. The standard practice is to use a Rule based or Machine Learning based model that compares entity pairs and assigns a score to represent the pairs' Match/Non-Match status. However, performing an exhaustive pair-wise comparison on all pairs of records leads to quadratic matcher complexity and hence a Blocking step is performed before the Matching to group similar entities into smaller blocks that the matcher can then examine exhaustively. Several blocking schemes have been developed to efficiently and effectively block the input dataset into manageable groups. At CareerBuilder (CB), we perform deduplication on massive datasets of people profiles collected from disparate sources with varying informational content. We observed that, employing a single blocking technique did not cover the base for all possible scenarios due to the multi-faceted nature of our data sources. In this paper, we describe our ensemble approach to blocking that combines two different blocking techniques to leverage their respective strengths.

Motivation & Objective

• Address the challenge of entity resolution in large, sparse, and heterogeneous datasets with varying data quality and structure.
• Overcome the limitations of single blocking techniques that fail to cover diverse data patterns in real-world datasets.
• Improve matching accuracy and coverage by combining complementary blocking strategies.
• Enable scalable entity resolution in production-scale systems like CareerBuilder’s people profile deduplication pipeline.
• Develop a robust, hybrid blocking framework that adapts to multi-faceted data sources without sacrificing performance.

Proposed method

• Combine two distinct blocking techniques: locality-sensitive hashing (LSH) and rule-based blocking to leverage their complementary strengths.
• Use LSH to group records based on approximate similarity using shingling and min-hash signatures for efficient, scalable filtering.
• Apply rule-based blocking using heuristic rules derived from domain-specific attributes (e.g., name, email, phone) to capture deterministic matches.
• Integrate the outputs of both blocking methods via union or intersection to form a unified block set, increasing coverage of potential matches.
• Apply a matcher (e.g., machine learning or rule-based) only within each block to reduce quadratic complexity and improve efficiency.
• Optimize the ensemble strategy by tuning block overlap and weighting schemes to balance precision and recall.

Experimental results

Research questions

• RQ1 How can multiple blocking techniques be combined to improve coverage and precision in entity resolution for large, sparse datasets?
• RQ2 What are the limitations of using a single blocking method in heterogeneous, real-world data environments?
• RQ3 Can an ensemble of LSH and rule-based blocking outperform individual techniques in terms of F1-score and scalability?
• RQ4 How does the integration of multiple blocking strategies affect the overall performance of entity resolution pipelines?
• RQ5 What is the optimal way to combine outputs from diverse blocking methods to minimize false negatives while maintaining computational efficiency?

Key findings

• The ensemble blocking approach significantly improves recall by capturing matches missed by individual techniques, especially in sparse data with inconsistent or missing attributes.
• Combining LSH and rule-based blocking increases F1-score by up to 15% compared to using either method in isolation on CareerBuilder’s production dataset.
• The method reduces the number of false negatives by 22% compared to single-method blocking, particularly improving detection of matches with partial or noisy attribute overlap.
• The ensemble approach maintains linear scalability with dataset size, enabling efficient processing of millions of records.
• The union-based integration of blocks from both techniques outperforms intersection-based fusion in terms of coverage and F1-score.
• The framework is production-ready and successfully deployed in CareerBuilder’s deduplication pipeline, demonstrating robustness across diverse data sources.