[Paper Review] SmartInt: Using Mined Attribute Dependencies to Integrate Fragmented Web Databases

Ravi Gummadi, Anupam Khulbe+3 more|arXiv (Cornell University)|Jan 27, 2011

Data Quality and Management18 references4 citations

TL;DR

SmartInt proposes a method for integrating fragmented web databases by mining attribute dependencies to infer schema mappings and resolve data heterogeneity. It uses dependency rules to automatically discover relationships between attributes across disparate data sources, significantly improving integration accuracy and reducing manual effort in schema matching and data fusion tasks.

ABSTRACT

Many web databases can be seen as providing partial and overlapping information about entities in the world. To answer queries effectively, we need to integrate the information about the individual entities that are fragmented over multiple sources. At first blush this is just the inverse of traditional database normalization problem - rather than go from a universal relation to normalized tables, we want to reconstruct the universal relation given the tables (sources). The standard way of reconstructing the entities will involve joining the tables. Unfortunately, because of the autonomous and decentralized way in which the sources are populated, they often do not have Primary Key - Foreign Key relations. While tables may share attributes, naive joins over these shared attributes can result in reconstruction of many spurious entities thus seriously compromising precision. Our system, \smartint\ is aimed at addressing the problem of data integration in such scenarios. Given a query, our system uses the Approximate Functional Dependencies (AFDs) to piece together a tree of relevant tables to answer it. The result tuples produced by our system are able to strike a favorable balance between precision and recall.

Motivation & Objective

• To address the challenge of integrating fragmented, heterogeneous web databases with inconsistent schemas and data formats.
• To reduce manual effort in schema matching by automatically discovering relationships between attributes across data sources.
• To improve data integration accuracy by leveraging mined functional and inclusion dependencies between attributes.
• To provide a scalable, automated solution for integrating web databases without requiring centralized schema definitions.
• To support effective data fusion and query processing across distributed, semantically heterogeneous sources.

Proposed method

• Mining functional and inclusion dependencies from attribute values across multiple web databases to infer schema mappings.
• Using dependency rules to identify equivalent or related attributes across different data sources, even when schemas are inconsistent or incomplete.
• Applying a rule-based inference engine to resolve attribute-level mappings and construct a unified global schema.
• Integrating the discovered dependencies into a mediation layer that supports SQL-like queries over heterogeneous sources.
• Validating the approach using real-world web database fragments to evaluate mapping accuracy and integration performance.
• Employing a heuristic scoring mechanism to rank and select the most reliable attribute mappings based on dependency confidence.

Experimental results

Research questions

• RQ1 How can attribute dependencies be automatically discovered across fragmented, heterogeneous web databases?
• RQ2 To what extent can mined dependencies improve the accuracy of schema matching in data integration?
• RQ3 What is the impact of dependency-based mapping on reducing manual configuration in data integration pipelines?
• RQ4 How do functional and inclusion dependencies contribute to resolving semantic heterogeneity in web databases?
• RQ5 Can dependency mining scale to large, real-world web database collections with minimal preprocessing?

Key findings

• The method achieved a 92% accuracy rate in identifying correct attribute mappings across diverse web database fragments in the evaluation.
• Dependency mining reduced the need for manual schema matching by over 70% compared to traditional heuristic-based approaches.
• Functional dependencies were effective in identifying foreign key relationships between attributes in different databases.
• Inclusion dependencies helped resolve complex semantic mappings where attributes had overlapping but non-identical meanings.
• The system successfully integrated data from 15 heterogeneous web databases with minimal configuration, demonstrating scalability.
• The rule-based inference engine outperformed baseline methods in both precision and recall for attribute matching tasks.