[Paper Review] Parallel Sorted Neighborhood Blocking with MapReduce

Lars Kolb, Andreas Thor+1 more|arXiv (Cornell University)|Oct 15, 2010

Data Quality and Management1 references17 citations

TL;DR

This paper proposes two MapReduce-based parallel implementations—JobSN and RepSN—for Sorted Neighborhood blocking in entity resolution, enabling scalable, efficient processing of large datasets. By using multiple MapReduce jobs or tailored data replication, the approaches achieve linear scalability and reduce execution time significantly compared to sequential methods, especially under data skew and varying window sizes.

ABSTRACT

Cloud infrastructures enable the efficient parallel execution of data-intensive tasks such as entity resolution on large datasets. We investigate challenges and possible solutions of using the MapReduce programming model for parallel entity resolution. In particular, we propose and evaluate two MapReduce-based implementations for Sorted Neighborhood blocking that either use multiple MapReduce jobs or apply a tailored data replication.

Motivation & Objective

• To enable efficient, scalable entity resolution on large datasets using the MapReduce model.
• To address the challenges of implementing Sorted Neighborhood blocking in a distributed MapReduce environment.
• To design and evaluate two distinct parallelization strategies—multi-job and data replication—for Sorted Neighborhood blocking.
• To analyze performance under data skew and varying window sizes in distributed execution.
• To demonstrate that MapReduce can effectively support complex data-intensive workflows like entity resolution.

Proposed method

• JobSN uses two MapReduce jobs: the first sorts and partitions entities by blocking key, the second re-partitions and applies sliding window matching across blocks.
• RepSN employs a single MapReduce job with custom map configuration and close functions to maintain sliding window state across mappers via in-memory replication.
• Both methods use composite keys (e.g., reducer_id.blocking_key) to ensure correct grouping and ordering for sliding window evaluation.
• The partitioning function maps blocking keys to reducers, ensuring load balancing and correct data shuffling across nodes.
• A sliding window of size w is applied in the reduce phase to generate candidate matches only between nearby entities in the sorted order.
• RepSN maintains a buffer of the last w−1 entities per reducer to enable window continuity across mapper boundaries, using min-heap logic for replacement.

Experimental results

Research questions

• RQ1 Can Sorted Neighborhood blocking be effectively parallelized using the MapReduce model for large-scale entity resolution?
• RQ2 How do different MapReduce-based designs (multi-job vs. data replication) compare in terms of performance and scalability?
• RQ3 What is the impact of data skew on the performance of parallel Sorted Neighborhood blocking in MapReduce?
• RQ4 How does window size affect execution time and load distribution in the proposed approaches?
• RQ5 Can the proposed methods maintain correctness and match quality while achieving linear speedup on large datasets?

Key findings

• JobSN achieves linear scalability with increasing data size and reduces execution time by up to 80% compared to the sequential approach on large datasets.
• RepSN outperforms JobSN in scenarios with high data skew, reducing execution time by up to 40% due to better load balancing through in-memory replication.
• Both approaches maintain high match quality, preserving recall and precision comparable to the sequential Sorted Neighborhood method.
• The performance of both methods degrades with larger window sizes, but RepSN shows more stable performance under varying window sizes.
• Data skew significantly impacts JobSN’s performance due to uneven reducer loads, while RepSN mitigates this through intelligent data replication and buffer management.
• The evaluation confirms that MapReduce is viable for complex entity resolution workflows when optimized with custom partitioning and window handling strategies.