[Paper Review] A Hybrid Data Cleaning Framework using Markov Logic Networks

Yunjun Gao, Congcong Ge+4 more|arXiv (Cornell University)|Mar 14, 2019

Data Quality and ManagementDecision Sciences27 references3 citations

TL;DR

This paper proposes MLNClean, a hybrid data cleaning framework that integrates Markov Logic Networks (MLNs) to jointly address schema-level and instance-level errors—such as integrity constraint violations, typos, replacement errors, and duplicates. By leveraging a two-stage process with MLN-based indexing, reliability scores, and fusion scores, MLNClean achieves superior accuracy and efficiency over state-of-the-art methods on real and synthetic datasets.

ABSTRACT

With the increase of dirty data, data cleaning turns into a crux of data analysis. Most of the existing algorithms rely on either qualitative techniques (e.g., data rules) or quantitative ones (e.g., statistical methods). In this paper, we present a novel hybrid data cleaning framework on top of Markov logic networks (MLNs), termed as MLNClean, which is capable of cleaning both schema-level and instance-level errors. MLNClean mainly consists of two cleaning stages, namely, first cleaning multiple data versions separately (each of which corresponds to one data rule), and then deriving the final clean data based on multiple data versions. Moreover, we propose a series of techniques/concepts, e.g., the MLN index, the concepts of reliability score and fusion score, to facilitate the cleaning process. Extensive experimental results on both real and synthetic datasets demonstrate the superiority of MLNClean to the state-of-the-art approach in terms of both accuracy and efficiency.

Motivation & Objective

• Address the limitations of existing data cleaning methods that rely solely on qualitative rules or quantitative statistics, which often fail to handle diverse error types simultaneously.
• Overcome the inefficiency of isolated error detection and repair processes by integrating both qualitative and quantitative techniques in a unified framework.
• Improve data cleaning accuracy and efficiency by combining integrity constraints with probabilistic reasoning via Markov Logic Networks.
• Enable robust, scalable data cleaning through a two-layer MLN index and distributed processing on Spark.
• Develop novel concepts—reliability score and fusion score—to enhance conflict resolution and data version integration during cleaning.

Proposed method

• Construct a two-layer MLN index to organize data into groups based on functional dependencies and conditional functional dependencies, enabling efficient error detection.
• Apply an AGP (Abnormal Group Processing) strategy to detect and clean groups of tuples violating integrity constraints using MLN inference.
• Introduce a reliability score to assess the trustworthiness of individual attribute values within each group, guiding repair decisions in the RSC (Reliability-based Score-based Cleaning) method.
• Use a fusion score to resolve conflicts when merging multiple cleaned data versions, ensuring consistency and minimizing redundant repairs.
• Implement a distributed version of MLNClean on Spark to scale across clusters, with load balancing across workers and efficient similarity computation using Levenshtein distance.
• Leverage Levenshtein distance as the primary metric for measuring string similarity in typo and replacement error detection, due to its sensitivity to character-level differences.

Experimental results

Research questions

• RQ1 Can a hybrid framework combining qualitative integrity constraints and quantitative probabilistic reasoning outperform existing data cleaning methods in accuracy and efficiency?
• RQ2 How effectively can Markov Logic Networks model and resolve complex, interdependent data errors across both schema-level and instance-level dimensions?
• RQ3 To what extent does the use of reliability and fusion scores improve conflict resolution and data consistency during multi-version data integration?
• RQ4 How does the choice of distance metric (e.g., Levenshtein vs. cosine) affect the detection of typos and replacement errors in the cleaning pipeline?
• RQ5 What is the scalability and performance gain of a distributed MLNClean implementation on large-scale datasets using Spark?

Key findings

• MLNClean achieves over 95% accuracy on both HAI and TPC-H datasets even at a 30% error rate, with a drop of less than 3% in accuracy compared to 5% error rate.
• The distributed MLNClean implementation achieves a 6.7x speedup when scaling from 2 to 10 workers on the TPC-H dataset, with minimal fluctuation in accuracy.
• Levenshtein distance outperforms cosine distance in error detection, as it better captures character-level differences critical for identifying typos and replacement errors.
• The FSCR algorithm achieves higher recall than precision due to more aggressive conflict detection, which is mitigated by the high reliability of prior cleaning stages.
• The two-stage cleaning pipeline—first cleaning multiple data versions independently, then fusing them using fusion scores—significantly improves final data quality compared to single-stage approaches.
• The MLN index enables efficient grouping and inference, reducing redundant computation and accelerating the overall cleaning process.