[Paper Review] Data Validation for Big Live Data

Malcolm Crowe, Carolyn E. Begg+2 more|arXiv (Cornell University)|Oct 5, 2021

Cloud Computing and Resource Management3 citations

TL;DR

This paper proposes a readCheck validator mechanism to ensure data timeliness and reduce network traffic in virtual data warehouses for big live data. By extending REST ETag-like row-version validation, it enables optimistic concurrency control with full ACID properties across distributed, heterogeneous data sources, ensuring up-to-date and verifiable query results without materializing data at the requester.

ABSTRACT

Data Integration of heterogeneous data sources relies either on periodically transferring large amounts of data to a physical Data Warehouse or retrieving data from the sources on request only. The latter results in the creation of what is referred to as a virtual Data Warehouse, which is preferable when the use of the latest data is paramount. However, the downside is that it adds network traffic and suffers from performance degradation when the amount of data is high. In this paper, we propose the use of a readCheck validator to ensure the timeliness of the queried data and reduced data traffic. It is further shown that the readCheck allows transactions to update data in the data sources obeying full Atomicity, Consistency, Isolation, and Durability (ACID) properties.

Motivation & Objective

• To address the challenge of maintaining up-to-date, consistent, and verifiable data in virtual data warehouses where data remains in distributed, heterogeneous sources.
• To reduce network traffic and performance degradation caused by frequent polling of live data sources in real-time integration scenarios.
• To enable distributed, ACID-compliant updates to data sources while ensuring data correctness and consistency across the global view.
• To support data curation and provenance tracking in regulatory and enterprise settings where data integrity and timeliness are critical.
• To provide a lightweight, interoperable validation mechanism that works across different data models and ownership boundaries without requiring data replication.

Proposed method

• The readCheck mechanism extends RFC 7232 ETags to include row-version identifiers (RVV) that track the transaction log position and data source table for each data row.
• Each query result row is annotated with an RVV string (e.g., 'Statistics:578:474') indicating the source table, log position, and transaction ID, enabling change detection.
• The requester uses the RVV to validate whether a previously retrieved result is still current; if unchanged, the result is reused, avoiding redundant queries.
• The system supports updatable REST views via PUT/POST/DELETE operations on the underlying data sources, enabling distributed data curation with ACID compliance.
• A Pyrrho database prototype demonstrates the mechanism by automatically generating REST views and managing RVV and ETag propagation across joined views.
• The approach assumes no data conflicts during transformation from individual views (Vi) to the global view (V), with future work planned on using metadata to automate schema mapping.

Experimental results

Research questions

• RQ1 How can data timeliness be validated in a virtual data warehouse without materializing data at the requester?
• RQ2 What mechanism enables efficient, low-overhead validation of query results to reduce redundant data transfer in live data integration?
• RQ3 Can a row-version validation system support optimistic concurrency control with full ACID properties across distributed, heterogeneous data sources?
• RQ4 How can data provenance and correctness be maintained when integrating live data from multiple independent, owned data sources?
• RQ5 To what extent can a REST-based, ETag-like mechanism be extended to support complex, joined views and distributed data updates?

Key findings

• The readCheck mechanism successfully enables validation of query results using row-version identifiers (RVV), allowing the requester to determine if a result is still current without re-querying.
• By reusing cached results when the RVV remains unchanged, the system significantly reduces network traffic and query latency in live data scenarios.
• The prototype implementation using Pyrrho demonstrates that updatable REST views can be generated automatically, supporting PUT, POST, and DELETE operations on the underlying data sources.
• The system maintains full ACID properties for distributed updates by using transactional log positions in the RVV to detect conflicts and ensure consistency.
• The mechanism is interoperable: the requester only needs to verify the RVV string, without understanding the internal structure of the source data or validation logic.
• The approach supports data curation and provenance transparency, as each result row carries metadata about its source and transactional state, enabling auditability and trust.