[Paper Review] Efficient Synthesis of Compact Deep Neural Networks

Wenhan Xia, Hongxu Yin+1 more|arXiv (Cornell University)|Apr 18, 2020

Advanced Neural Network Applications47 references4 citations

TL;DR

This paper presents a grow-and-prune synthesis paradigm for efficiently generating compact, accurate deep neural networks (DNNs) and LSTMs, particularly for resource-constrained edge devices. By incrementally growing network connections with gradient guidance and then pruning low-magnitude weights, the method reduces training costs by up to 67% compared to retraining from scratch, while maintaining high accuracy in data-free and incremental learning scenarios using techniques like DeepInversion for synthetic data generation.

ABSTRACT

Deep neural networks (DNNs) have been deployed in myriad machine learning applications. However, advances in their accuracy are often achieved with increasingly complex and deep network architectures. These large, deep models are often unsuitable for real-world applications, due to their massive computational cost, high memory bandwidth, and long latency. For example, autonomous driving requires fast inference based on Internet-of-Things (IoT) edge devices operating under run-time energy and memory storage constraints. In such cases, compact DNNs can facilitate deployment due to their reduced energy consumption, memory requirement, and inference latency. Long short-term memories (LSTMs) are a type of recurrent neural network that have also found widespread use in the context of sequential data modeling. They also face a model size vs. accuracy trade-off. In this paper, we review major approaches for automatically synthesizing compact, yet accurate, DNN/LSTM models suitable for real-world applications. We also outline some challenges and future areas of exploration.

Motivation & Objective

• To address the challenge of deploying large, computationally expensive DNNs and LSTMs on resource-constrained edge devices such as IoT sensors and mobile platforms.
• To reduce the high training cost and memory footprint associated with retraining large models from scratch when new data arrive.
• To enable efficient model compression and incremental learning without requiring access to original training data, preserving privacy.
• To develop a data-free knowledge transfer framework that leverages trained models to synthesize high-fidelity training data for pruning and fine-tuning.
• To accelerate DNN training and model synthesis through hardware-aware, automated architecture search and pruning techniques.

Proposed method

• The grow-and-prune synthesis paradigm incrementally expands a DNN by adding new connections in a gradient-directed manner when new data arrive, enhancing learning capacity.
• After growth, the framework prunes low-magnitude weights to produce a compact, accurate model, reducing model size and inference latency.
• DeepInversion is used to generate high-fidelity, class-conditional synthetic images from a trained classifier without access to original data, using batch norm statistics and feature regularization.
• Adaptive DeepInversion enhances image diversity by introducing competition regularization between a teacher and student network during inversion.
• The method enables data-free knowledge transfer, model pruning, and incremental learning using only the trained model and no original training data.
• The approach is extended to various architectures, including ResNet-18, DeepSpeech2, and can be adapted to LSTMs, 3D CNNs, and language models for data-free training and compression.

Experimental results

Research questions

• RQ1 How can we efficiently synthesize compact DNNs and LSTMs that maintain high accuracy while reducing computational and memory costs for edge deployment?
• RQ2 What techniques enable effective incremental learning without retraining from scratch, especially when original training data are unavailable?
• RQ3 To what extent can a trained DNN generate high-fidelity synthetic data that preserves the original data distribution for knowledge transfer and model compression?
• RQ4 Can data-free pruning and model compression be achieved with minimal performance degradation using only the trained model and no access to training data?
• RQ5 How can hardware-aware model design and optimization techniques be integrated into automated DNN synthesis pipelines for real-world deployment?

Key findings

• The grow-and-prune framework reduced training cost by 64% for ResNet-18 on ImageNet and 67% for DeepSpeech2 on AN4 compared to retraining from scratch.
• DeepInversion successfully generated high-fidelity, diverse images across all classes using only a trained classifier, matching the performance of BigGAN without requiring GAN training on original data.
• The data-free pruning approach via DeepInversion achieved state-of-the-art results on CIFAR-10 and ImageNet, demonstrating effective knowledge transfer without original data.
• Adaptive DeepInversion improved image diversity through competition regularization between teacher and student networks, enhancing the quality of synthetic data.
• The method enabled efficient incremental learning with significant cost savings—60% for ResNet-18 and 62% for DeepSpeech2—while preserving model accuracy.
• The framework is extensible to diverse architectures, including LSTMs and 3D CNNs, enabling data-free synthesis for video, speech, text, and language modeling tasks.