[Paper Review] zkDL: Efficient Zero-Knowledge Proofs of Deep Learning Training

Hao-Chen Sun, Bai, Tonghe+2 more|arXiv (Cornell University)|Jul 30, 2023

Parallel Computing and Optimization Techniques4 citations

TL;DR

zkDL presents the first efficient zero-knowledge proof system for verifiable deep learning training, introducing zkReLU to natively handle ReLU activations and backpropagation without polynomial approximations. It achieves sub-second proof generation (under 1 second per batch) for 8-layer networks with 10M parameters and batch size 64, enabling privacy-preserving, scalable verification of model training on real-world deep neural networks.

ABSTRACT

The recent advancements in deep learning have brought about significant changes in various aspects of people's lives. Meanwhile, these rapid developments have raised concerns about the legitimacy of the training process of deep neural networks. To protect the intellectual properties of AI developers, directly examining the training process by accessing the model parameters and training data is often prohibited for verifiers. In response to this challenge, we present zero-knowledge deep learning (zkDL), an efficient zero-knowledge proof for deep learning training. To address the long-standing challenge of verifiable computations of non-linearities in deep learning training, we introduce zkReLU, a specialized proof for the ReLU activation and its backpropagation. zkReLU turns the disadvantage of non-arithmetic relations into an advantage, leading to the creation of FAC4DNN, our specialized arithmetic circuit design for modelling neural networks. This design aggregates the proofs over different layers and training steps, without being constrained by their sequential order in the training process. With our new CUDA implementation that achieves full compatibility with the tensor structures and the aggregated proof design, zkDL enables the generation of complete and sound proofs in less than a second per batch update for an 8-layer neural network with 10M parameters and a batch size of 64, while provably ensuring the privacy of data and model parameters. To our best knowledge, we are not aware of any existing work on zero-knowledge proof of deep learning training that is scalable to million-size networks.

Motivation & Objective

• To address the lack of scalable, privacy-preserving verification mechanisms for deep learning training in industrial and regulatory contexts.
• To overcome the fundamental challenge of verifying non-arithmetic operations—particularly ReLU activations—within zero-knowledge proof systems.
• To design an arithmetic circuit model that efficiently captures the layered, sequential, and repetitive structure of deep learning training across multiple steps and layers.
• To enable end-to-end verifiable training for large-scale deep neural networks, ensuring both correctness and confidentiality of data and model parameters.
• To develop a CUDA-optimized implementation that achieves practical performance for real-world deployment of zero-knowledge verifiable training.

Proposed method

• Introduces zkReLU, a specialized zero-knowledge proof for ReLU and its backpropagation, leveraging the inherent non-linearity as a computational advantage rather than a constraint.
• Proposes FAC4DNN, an arithmetic circuit design that models the entire deep learning training process—including forward and backward passes, layer-wise operations, and batch processing—while preserving tensor and layer structures.
• Employs proof aggregation across layers and training steps by exploiting structural symmetries and computational patterns in zkReLU, enabling non-sequential proof composition independent of training order.
• Designs a CUDA-native implementation that aligns with deep learning frameworks’ tensor layouts, enabling high-performance proof generation and verification on GPU hardware.
• Uses a zero-knowledge proof backend compatible with modern ZKP systems, ensuring soundness and zero-knowledge guarantees for both training data and model parameters.
• Integrates the proof system into a full training pipeline, allowing model developers to generate proofs after each batch update without modifying the training logic.

Experimental results

Research questions

• RQ1 Can non-arithmetic operations like ReLU be natively verified in zero-knowledge proofs without relying on polynomial approximations?
• RQ2 How can proof generation for deep learning training be made efficient enough for large-scale models with millions of parameters?
• RQ3 Can proofs across different layers and training steps be aggregated in a way that bypasses sequential dependencies and reduces computational overhead?
• RQ4 Is it feasible to build a GPU-optimized zero-knowledge proof system for deep learning training that achieves sub-second proof generation times?
• RQ5 Can such a system ensure both correctness of training and strong privacy guarantees for both training data and model parameters?

Key findings

• zkDL generates complete and sound zero-knowledge proofs for deep learning training in under one second per batch update for an 8-layer network with 10 million parameters and a batch size of 64.
• The system supports verifiable training for real-world-scale deep neural networks, with no prior work achieving similar scalability to million-parameter models.
• Proof aggregation via FAC4DNN enables computational and communication overhead reductions by decoupling proof composition from the sequential order of training steps and layers.
• The CUDA-optimized implementation achieves full compatibility with standard deep learning tensor structures, enabling efficient integration into existing training pipelines.
• Verification time is comparable to proving time due to the sequential nature of the verification protocol, highlighting a need for future parallelized verification algorithms.
• zkDL is the first framework to provide end-to-end zero-knowledge verifiability for deep learning training with provable privacy and performance at scale.