[论文解读] Entity Consolidation: The Golden Record Problem.

Dong Deng, Wenbo Tao+7 more|arXiv (Cornell University)|Sep 29, 2017

Data Quality and Management被引用 3

一句话总结

本文提出了一种可扩展的实体合并算法，通过从聚类中挖掘匹配规则、将规则分组以供人工审查，并应用真相发现技术解决冲突，从而实现黄金记录的自动化生成。通过仅验证100组规则，该方法实现了98%的精确率和75%的召回率，当与多数共识结合时，黄金记录的精确率提高了40%。

ABSTRACT

Four key processes in data integration are: data preparation (i.e., extracting, transforming, and cleaning data), schema integration (i.e., identifying attribute correspondences), entity resolution (i.e., finding clusters of records that represent the same entity) and entity consolidation (i.e., merging each cluster into a 'golden record', which contains the canonical value for each attribute). In real-world scenarios, Master Data Management (MDM) is often employed to perform the last two steps using a 'match-merge' module, which is driven by a collection of user-written match and merge rules. However, it is well understood that such solutions often fail to scale to the sizes and complexity of problems currently being addressed. Scalable solutions to entity resolution have been previously developed to generate a collection of clusters of records representing the same entity. In this paper, we propose a scalable entity consolidation algorithm to merge these clusters into 'golden records'. We first automatically generate matching rules from the clusters and then group these rules into sets with common characteristics to cut down on the number which must be verified by a human. Next, for the human-approved rule groups, we apply them to merge the duplicate attribute values in the clusters. Finally, we employ existing truth discovery methods, such as majority consensus (MC), to resolve remaining conflicts in the clusters. We applied our methods on three real-world datasets. In a dataset with 31,023 clusters and 80,451 duplicate records, 72,239 matching rules were generated. By having a human con rm only 100 algorithm-generated rule groups, we achieved a recall of 75% and a precision of 98% for merging duplicate attribute values. When we invoked our algorithm prior to running MC, we improved the precision of golden record construction by 40%.

研究动机与目标

• 解决传统匹配-合并模块在主数据管理（MDM）中实体合并时的可扩展性限制。
• 通过基于共享特征对自动生成的匹配规则进行分组，减少人工验证的工作量。
• 通过整合自动化规则应用与真相发现技术，提高黄金记录构建的准确性。
• 在包含大规模重复记录的真实数据集上评估所提方法的有效性。

提出的方法

• 从包含80,451条重复记录的31,023个聚类中自动生成72,239条匹配规则。
• 基于共同特征将生成的规则分组，以减少需要人工验证的规则数量。
• 将人工批准的规则组应用于聚类内重复属性值的合并。
• 采用多数共识（MC）作为真相发现方法，解决剩余的属性值冲突。
• 采用结合规则基础合并与共识基础解决的混合方法，实现稳健的黄金记录构建。

实验结果

研究问题

• RQ1 自动化规则生成与分组是否能在保持高精确度的同时减少实体合并中的人工工作量？
• RQ2 在真相发现用于黄金记录构建之前，使用自动化规则进行预处理能在多大程度上提升其性能？
• RQ3 该方法在具有复杂重复模式的大规模真实数据集上是否具备良好的可扩展性？
• RQ4 在仅使用有限数量的人工验证规则组时，精确率与召回率之间的权衡如何？

主要发现

• 该方法在真实世界数据集中，从包含80,451条重复记录的31,023个聚类中生成了72,239条匹配规则。
• 通过仅验证100个人工批准的规则组，该方法在合并重复属性值时实现了98%的精确率和75%的召回率。
• 在多数共识之前集成该算法，使黄金记录构建的精确率提高了40%。
• 规则分组策略显著减少了需要手动验证的规则数量，从而提升了可扩展性。