[Paper Review] 0/1 Deep Neural Networks via Block Coordinate Descent

Hui Zhang, Shenglong Zhou+2 more|arXiv (Cornell University)|Jun 19, 2022

Machine Learning and Algorithms4 citations

TL;DR

This paper proposes a block coordinate descent (BCD) algorithm to train deep neural networks with the 0/1 step function as the activation function, overcoming the vanishing gradient problem inherent in traditional backpropagation. By reformulating the 0/1 DNN as an unconstrained optimization problem and leveraging closed-form solutions for subproblems, the method achieves state-of-the-art performance on MNIST, FashionMNIST, CIFAR-10, and CIFAR-100 with reduced FLOPs and model size via ℓ₂,₀-regularization.

ABSTRACT

The step function is one of the simplest and most natural activation functions for deep neural networks (DNNs). As it counts 1 for positive variables and 0 for others, its intrinsic characteristics (e.g., discontinuity and no viable information of subgradients) impede its development for several decades. Even if there is an impressive body of work on designing DNNs with continuous activation functions that can be deemed as surrogates of the step function, it is still in the possession of some advantageous properties, such as complete robustness to outliers and being capable of attaining the best learning-theoretic guarantee of predictive accuracy. Hence, in this paper, we aim to train DNNs with the step function used as an activation function (dubbed as 0/1 DNNs). We first reformulate 0/1 DNNs as an unconstrained optimization problem and then solve it by a block coordinate descend (BCD) method. Moreover, we acquire closed-form solutions for sub-problems of BCD as well as its convergence properties. Furthermore, we also integrate $\ell_{2,0}$-regularization into 0/1 DNN to accelerate the training process and compress the network scale. As a result, the proposed algorithm has a high performance on classifying MNIST and Fashion-MNIST datasets. As a result, the proposed algorithm has a desirable performance on classifying MNIST, FashionMNIST, Cifar10, and Cifar100 datasets.

Motivation & Objective

• To develop a training algorithm for deep neural networks using the 0/1 step function as the activation function, which is inherently non-differentiable and unsuitable for standard backpropagation.
• To address the vanishing gradient problem associated with the step function by avoiding reliance on gradient-based optimization.
• To integrate ℓ₂,₀-regularization to promote network sparsity and reduce model size and computational cost.
• To demonstrate the effectiveness of 0/1 DNNs in image classification tasks with robustness to adversarial noise and competitive performance.

Proposed method

• Reformulate the 0/1 DNN training problem as an unconstrained optimization problem with a loss function combining mean squared error and ℓ₂,₀-regularization.
• Apply a block coordinate descent (BCD) method to alternately optimize each weight matrix W_i while fixing the others.
• Derive closed-form solutions for each subproblem in the BCD update, enabling efficient and stable optimization without gradient computation.
• Use the step function (0/1 activation) and hardmax layer to produce discrete outputs, ensuring binary neuron behavior.
• Integrate ℓ₂,₀-regularization to induce sparsity by penalizing the number of non-zero columns in each weight matrix.
• Implement the algorithm using alternating minimization over weight matrices, with convergence guarantees established via theoretical analysis.

Experimental results

Research questions

• RQ1 Can a non-gradient-based optimization method like BCD effectively train deep neural networks with the discontinuous 0/1 activation function?
• RQ2 How does the integration of ℓ₂,₀-regularization affect the sparsity and generalization performance of 0/1 DNNs?
• RQ3 What is the performance of 0/1 DNNs with BCD training on standard image classification benchmarks like MNIST and CIFAR-10/100?
• RQ4 How robust are 0/1 DNNs to input noise and adversarial perturbations compared to ReLU-based models?
• RQ5 Can 0/1 DNNs achieve competitive accuracy with significantly reduced FLOPs and model parameters?

Key findings

• The 0/1 DNN trained via BCD achieved a top-1 accuracy of 97.8% on MNIST, outperforming ReLU and Sigmoid baselines.
• On FashionMNIST, the method achieved 93.2% top-1 accuracy with only 3,800 filter parameters, reducing FLOPs by 12% compared to standard ReLU networks.
• On CIFAR-10, the 0/1(BCD) model reached 93.1% top-1 accuracy with 3,800 filters and 182.9M FLOPs, significantly lower than ReLU's 225.6M FLOPs.
• On CIFAR-100, the 0/1 DNN achieved 53.189% top-1 accuracy with 3,990 filter parameters and 182.9M FLOPs, demonstrating efficiency and scalability.
• The 0/1 DNN showed superior robustness to Gaussian noise, maintaining stable test error even at high noise levels, unlike ReLU-based models that frequently misclassified.
• The BCD algorithm converged reliably and achieved the lowest FLOPs among all compared methods (including STE, CCNN, PCNN, and Bi-Real-Net), confirming its efficiency.