[Paper Review] Auditable data structures: theory and applications

Andrea Canciani, Claudio Felicioli+2 more|arXiv (Cornell University)|Jun 2, 2023

Security and Verification in Computing4 citations

TL;DR

This paper introduces a formal framework for externally auditable data structures that enable trust in data history consistency without exposing sensitive data. By combining cryptographic digests, consistency proofs, and a notarization model on public blockchains, it allows external auditors to verify the integrity and sequential correctness of data operations while preserving data secrecy, as demonstrated in a hybrid blockchain implementation.

ABSTRACT

Every digital process needs to consume some data in order to work properly. It is very common for applications to use some external data in their processes, getting them by sources such as external APIs. Therefore, trusting the received data becomes crucial in such scenarios, considering that if the data are not self-produced by the consumer, the trust in the external data source, or in the data that the source produces, can not always be taken for granted. The most used approach to generate trust in the external source is based on authenticated data structures, that are able to authenticate the source when queried through the generation of proofs. Such proofs are useful to assess authenticity or integrity, however, an external user could also be interested in verifying the data history and its consistency. This problem seems to be unaddressed by current literature, which proposes some approaches aimed at executing audits by internal actors with prior knowledge about the data structures. In this paper, we address the scenario of an external auditor with no data knowledge that wants to verify the data history consistency. We analyze the terminology and the current state of the art of the auditable data structures, then we will propose a general framework to support external audits from both internal and external users.

Motivation & Objective

• To address the lack of standardized terminology and frameworks for auditing data structure history consistency.
• To enable external auditors—without prior knowledge of data or data structures—to verify the consistency of data operations over time.
• To preserve data secrecy while allowing verification of data history through cryptographic proofs and notarization on public blockchains.
• To extend authenticated data structures to support external auditability of operational sequences, not just data integrity.
• To provide a formal foundation for comparing and evaluating auditable data structure solutions in real-world applications.

Proposed method

• Defining core terminology such as data structure operation, digest, consistency, and history to standardize the field.
• Introducing the concept of externally auditable data structures that use one-way data indirections to hide content while enabling proof verification.
• Proposing a framework where a notarization system publishes digest sequences and consistency proofs on a public blockchain to ensure external auditability.
• Employing a ledgers tree data structure to compress and verify the history of multiple private ledgers efficiently on a public blockchain.
• Using cryptographic hash functions with second-preimage resistance to ensure digest security and prevent forgery.
• Designing consistency proofs that allow external auditors to verify that a newer data structure version is derived from an older one via valid edit operations.

Figure 1: Scheme of the three-party model for data replication

Experimental results

Research questions

• RQ1 How can data structure history consistency be audited by an external auditor with no prior knowledge of the data or data structure?
• RQ2 What formal terminology and definitions are needed to standardize research and comparison in auditable data structures?
• RQ3 How can data secrecy be preserved while still enabling external verification of operational history?
• RQ4 What architectural components are required to support scalable, secure, and efficient external audit of data structure evolution?
• RQ5 How can existing systems like private DLTs or certificate revocation lists be enhanced using this framework?

Key findings

• The paper establishes a formal terminology set that clarifies key concepts such as data structure consistency, history, and consistency proofs, enabling clearer comparison across solutions.
• The proposed framework enables external auditors to verify data structure history consistency without accessing raw data, relying solely on digests and cryptographic proofs.
• The implementation in Traent's hybrid blockchain system demonstrates that a single public blockchain notarization can verify the consistency of multiple private ledger histories via a ledgers tree.
• Consistency proofs between successive digests allow auditors to detect forks or invalid operations, ensuring the uniqueness and correctness of data evolution.
• The use of public blockchain notarization ensures long-term auditability and immutability of data history, even when the private network is untrusted.
• The framework supports real-world applications such as auditable digital product passports, certificate management, and private DLTs with verifiable operational histories.

Figure 2: Scheme of the two-party model for data outsourcing