[Paper Review] Entity Consolidation: The Golden Record Problem.

Dong Deng, Wenbo Tao+7 more|arXiv (Cornell University)|Sep 29, 2017

Data Quality and Management3 citations

TL;DR

This paper proposes a scalable entity consolidation algorithm that automates golden record generation by mining matching rules from clusters, grouping them for human review, and applying truth discovery to resolve conflicts. By validating only 100 rule groups, the method achieves 98% precision and 75% recall, improving golden record precision by 40% when combined with majority consensus.

ABSTRACT

Four key processes in data integration are: data preparation (i.e., extracting, transforming, and cleaning data), schema integration (i.e., identifying attribute correspondences), entity resolution (i.e., finding clusters of records that represent the same entity) and entity consolidation (i.e., merging each cluster into a 'golden record', which contains the canonical value for each attribute). In real-world scenarios, Master Data Management (MDM) is often employed to perform the last two steps using a 'match-merge' module, which is driven by a collection of user-written match and merge rules. However, it is well understood that such solutions often fail to scale to the sizes and complexity of problems currently being addressed. Scalable solutions to entity resolution have been previously developed to generate a collection of clusters of records representing the same entity. In this paper, we propose a scalable entity consolidation algorithm to merge these clusters into 'golden records'. We first automatically generate matching rules from the clusters and then group these rules into sets with common characteristics to cut down on the number which must be verified by a human. Next, for the human-approved rule groups, we apply them to merge the duplicate attribute values in the clusters. Finally, we employ existing truth discovery methods, such as majority consensus (MC), to resolve remaining conflicts in the clusters. We applied our methods on three real-world datasets. In a dataset with 31,023 clusters and 80,451 duplicate records, 72,239 matching rules were generated. By having a human con rm only 100 algorithm-generated rule groups, we achieved a recall of 75% and a precision of 98% for merging duplicate attribute values. When we invoked our algorithm prior to running MC, we improved the precision of golden record construction by 40%.

Motivation & Objective

• To address the scalability limitations of traditional match-merge modules in Master Data Management (MDM) for entity consolidation.
• To reduce human effort in rule verification by grouping automatically generated matching rules based on shared characteristics.
• To improve the accuracy of golden record construction by integrating automated rule application with truth discovery techniques.
• To evaluate the effectiveness of the proposed method on real-world datasets with large-scale duplicate records.

Proposed method

• Automatically generate 72,239 matching rules from 31,023 clusters containing 80,451 duplicate records.
• Group the generated rules into sets based on common characteristics to reduce the number requiring human validation.
• Apply human-approved rule groups to merge duplicate attribute values within clusters.
• Employ majority consensus (MC) as a truth discovery method to resolve remaining attribute value conflicts.
• Use a hybrid approach combining rule-based merging and consensus-based resolution for robust golden record construction.

Experimental results

Research questions

• RQ1 Can automated rule generation and grouping reduce human effort in entity consolidation while maintaining high precision?
• RQ2 To what extent does pre-processing with automated rules improve the performance of truth discovery in golden record construction?
• RQ3 How effective is the proposed method in scaling to large, real-world datasets with complex duplication patterns?
• RQ4 What is the trade-off between recall and precision when using a limited number of human-verified rule groups?

Key findings

• The method generated 72,239 matching rules from 31,023 clusters containing 80,451 duplicate records in a real-world dataset.
• By validating only 100 human-approved rule groups, the approach achieved 98% precision and 75% recall in merging duplicate attribute values.
• The integration of the algorithm prior to majority consensus improved the precision of golden record construction by 40%.
• The rule grouping strategy significantly reduced the number of rules requiring manual verification, enhancing scalability.