[Paper Review] Ising-Dropout: A Regularization Method for Training and Compression of Deep Neural Networks

Hojjat Salehinejad, Shahrokh Valaee|arXiv (Cornell University)|Feb 7, 2019

Machine Learning and Data Classification10 references4 citations

TL;DR

This paper proposes Ising-Dropout, a regularization and model compression technique that uses the Ising model's energy minimization to adaptively drop visible and hidden units in deep neural networks during training. By mapping neuron activations to Ising weights and optimizing via the Fujitsu Digital Annealer, the method reduces parameter optimization by up to 55.86% on Fashion-MNIST while maintaining competitive classification accuracy.

ABSTRACT

Overfitting is a major problem in training machine learning models, specifically deep neural networks. This problem may be caused by imbalanced datasets and initialization of the model parameters, which conforms the model too closely to the training data and negatively affects the generalization performance of the model for unseen data. The original dropout is a regularization technique to drop hidden units randomly during training. In this paper, we propose an adaptive technique to wisely drop the visible and hidden units in a deep neural network using Ising energy of the network. The preliminary results show that the proposed approach can keep the classification performance competitive to the original network while eliminating optimization of unnecessary network parameters in each training cycle. The dropout state of units can also be applied to the trained (inference) model. This technique could compress the network in terms of number of parameters up to 41.18% and 55.86% for the classification task on the MNIST and Fashion-MNIST datasets, respectively.

Motivation & Objective

• To address overfitting in deep neural networks trained on imbalanced or limited datasets.
• To reduce the number of parameters optimized during training, accelerating convergence and improving generalization.
• To enable model compression by retaining only the most relevant, well-trained parameters in the inference phase.
• To explore the use of combinatorial optimization (via Ising model) for adaptive, non-random dropout in DNNs.
• To evaluate the trade-off between model compression and classification accuracy in deep MLPs on MNIST and Fashion-MNIST.

Proposed method

• Map the activations of each neuron in a deep MLP to a cost (Ising weight) in a spin glass model.
• Use the Fujitsu Digital Annealer—a specialized hardware for solving combinatorial optimization problems via MCMC—to minimize the Ising energy and determine optimal dropout states.
• Generate a binary mask (state vector) from the Ising optimizer that identifies which units to drop during training and inference.
• Apply the mask to the network’s weight tensors during backpropagation and inference, enabling dynamic parameter pruning.
• Integrate the Ising-Dropout process into each mini-batch training cycle, re-optimizing the mask at every epoch.
• Use the same mask for both training and inference, enabling model compression without retraining.

Experimental results

Research questions

• RQ1 Can Ising energy minimization produce a more effective, adaptive dropout strategy than random dropout in deep neural networks?
• RQ2 To what extent can Ising-Dropout reduce the number of optimized parameters during training while maintaining or improving generalization?
• RQ3 How does the performance of Ising-Dropout compare to standard dropout and no dropout in terms of accuracy and model compression?
• RQ4 Can the Ising-Dropout mask be reused at inference time to achieve significant model compression without performance degradation?
• RQ5 What is the trade-off between model size reduction and classification accuracy when applying Ising-Dropout across different network depths?

Key findings

• Ising-Dropout achieved a 41.18% reduction in model parameters on the MNIST dataset while maintaining competitive classification accuracy.
• On the more complex Fashion-MNIST dataset, the method compressed the model by 55.86% with only a 5.84% drop in accuracy compared to the full network.
• The method outperformed random dropout in both accuracy and compression efficiency, especially in deeper networks.
• The inference model was compressed by up to 55.86% without retraining, preserving performance through adaptive selection of well-trained parameters.
• The technique maintained high performance even with significant parameter pruning, demonstrating effective regularization and generalization.
• Visualizations showed that Ising-Dropout preserved essential input features (e.g., digit shapes) while removing background or redundant pixels.