[Paper Review] DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training

Aochuan Chen, Yimeng Zhang+8 more|arXiv (Cornell University)|Oct 3, 2023

Adversarial Robustness in Machine LearningComputer Science3 citations

TL;DR

DeepZero introduces a scalable zeroth-order (ZO) optimization framework for training deep neural networks (DNNs) from scratch, overcoming prior limitations in accuracy and efficiency. By leveraging coordinate-wise gradient estimation, sparsity-induced pruning, and feature reuse with forward parallelization, DeepZero achieves state-of-the-art accuracy on ResNet-20 for CIFAR-10, approaching first-order training performance while enabling practical black-box applications in adversarial defense and PDE error correction.

ABSTRACT

Zeroth-order (ZO) optimization has become a popular technique for solving machine learning (ML) problems when first-order (FO) information is difficult or impossible to obtain. However, the scalability of ZO optimization remains an open problem: Its use has primarily been limited to relatively small-scale ML problems, such as sample-wise adversarial attack generation. To our best knowledge, no prior work has demonstrated the effectiveness of ZO optimization in training deep neural networks (DNNs) without a significant decrease in performance. To overcome this roadblock, we develop DeepZero, a principled ZO deep learning (DL) framework that can scale ZO optimization to DNN training from scratch through three primary innovations. First, we demonstrate the advantages of coordinatewise gradient estimation (CGE) over randomized vector-wise gradient estimation in training accuracy and computational efficiency. Second, we propose a sparsityinduced ZO training protocol that extends the model pruning methodology using only finite differences to explore and exploit the sparse DL prior in CGE. Third, we develop the methods of feature reuse and forward parallelization to advance the practical implementations of ZO training. Our extensive experiments show that DeepZero achieves state-of-the-art (SOTA) accuracy on ResNet-20 trained on CIFAR-10, approaching FO training performance for the first time. Furthermore, we show the practical utility of DeepZero in applications of certified adversarial defense and DL-based partial differential equation error correction, achieving 10-20% improvement over SOTA. We believe our results will inspire future research on scalable ZO optimization and contribute to advancing DL with black box. Codes are available at https://github.com/OPTML-Group/DeepZero.

Motivation & Objective

• Address the scalability gap in zeroth-order (ZO) optimization for deep learning, where prior methods fail to train large models effectively.
• Overcome the high bias and computational cost of finite-difference-based ZO gradient estimation in high-dimensional DNNs.
• Enable end-to-end ZO training of deep models without access to computational graphs or backpropagation, supporting black-box and hardware-constrained settings.
• Demonstrate practical utility of ZO training in real-world applications such as certified adversarial defense and physics-informed PDE error correction.
• Achieve performance close to first-order methods in accuracy while maintaining query efficiency and model interpretability through sparsity.

Proposed method

• Adopt coordinate-wise gradient estimation (CGE) instead of randomized vector-wise estimation (RGE), reducing variance and improving convergence in deep networks.
• Introduce a sparsity-induced ZO training protocol that leverages model pruning to exploit the inherent sparsity of deep neural networks during finite-difference gradient estimation.
• Implement feature reuse and forward parallelization to reduce redundant forward passes and accelerate ZO training across multiple parameters.
• Design a computational graph-free training pipeline that relies solely on model queries, enabling deployment in black-box or resource-constrained environments.
• Integrate the ZO optimization framework within the Adam optimizer, enabling adaptive learning rates and stable training dynamics.
• Apply the framework to both supervised learning (e.g., CIFAR-10) and physics-informed learning (e.g., PDE error correction) via a unified loss formulation.

Figure 1: Overview of our DeepZero framework. A: ZO gradient estimation via model queries (Sec. 3 ). B: Model pruning guides gradient sparsity (Sec. 4 ). C: Acceleration by parallelization and feature reuse (Sec. 5 ). D: DeepZero comparison with the computational graph free baseline Pattern Search (

Experimental results

Research questions

• RQ1 Can coordinate-wise gradient estimation (CGE) outperform vector-wise randomized estimation (RGE) in terms of training accuracy and computational efficiency for deep ZO optimization?
• RQ2 How can model sparsity be exploited to improve the convergence and stability of ZO training in high-dimensional DNNs?
• RQ3 To what extent can feature reuse and forward parallelization reduce the query cost of ZO optimization in large-scale DNN training?
• RQ4 Can ZO optimization achieve performance comparable to first-order methods in training deep models from scratch?
• RQ5 Can the proposed ZO framework be effectively applied to practical black-box learning tasks such as certified adversarial defense and PDE error correction?

Key findings

• DeepZero achieves state-of-the-art accuracy on ResNet-20 trained on CIFAR-10 using ZO optimization, approaching the performance of first-order training methods.
• Coordinate-wise gradient estimation (CGE) significantly improves training accuracy and reduces computational cost compared to randomized vector-wise estimation, especially in deeper networks.
• The sparsity-induced training protocol enables stable and efficient ZO training by exploiting the inherent sparsity of deep models during finite-difference estimation.
• Feature reuse and forward parallelization reduce redundant model queries, accelerating ZO training by up to 3.5× in practical settings.
• DeepZero achieves 10–20% improvement over state-of-the-art methods in certified adversarial defense and physics-informed PDE error correction tasks.
• The framework demonstrates the feasibility of end-to-end ZO training of deep models without computational graphs or backpropagation, enabling deployment in black-box and on-chip learning scenarios.

Figure A1: Illustration of the simple CNN considered with different depths.