[Paper Review] Exploring the Imposition of Synaptic Precision Restrictions For Evolutionary Synthesis of Deep Neural Networks

Mohammad Javad Shafiee, Francis Li+1 more|arXiv (Cornell University)|Jul 1, 2017

Evolutionary Algorithms and Applications7 references3 citations

TL;DR

This paper proposes an evolutionary synthesis framework for deep neural networks that enforces 16-bit synaptic precision constraints across generations, using a stochastic, heredity-inspired probabilistic model to evolve sparser, faster networks. The approach achieves a tenfold reduction in synapses and a 30+ fps inference speedup (from 7 to 37 fps) on the Parse-27k dataset while maintaining consistent precision and recall.

ABSTRACT

A key contributing factor to incredible success of deep neural networks has been the significant rise on massively parallel computing devices allowing researchers to greatly increase the size and depth of deep neural networks, leading to significant improvements in modeling accuracy. Although deeper, larger, or complex deep neural networks have shown considerable promise, the computational complexity of such networks is a major barrier to utilization in resource-starved scenarios. We explore the synaptogenesis of deep neural networks in the formation of efficient deep neural network architectures within an evolutionary deep intelligence framework, where a probabilistic generative modeling strategy is introduced to stochastically synthesize increasingly efficient yet effective offspring deep neural networks over generations, mimicking evolutionary processes such as heredity, random mutation, and natural selection in a probabilistic manner. In this study, we primarily explore the imposition of synaptic precision restrictions and its impact on the evolutionary synthesis of deep neural networks to synthesize more efficient network architectures tailored for resource-starved scenarios. Experimental results show significant improvements in synaptic efficiency (~10X decrease for GoogLeNet-based DetectNet) and inference speed (>5X increase for GoogLeNet-based DetectNet) while preserving modeling accuracy.

Motivation & Objective

• To investigate the impact of imposing synaptic precision restrictions on the evolutionary synthesis of deep neural networks.
• To address the challenge of reducing computational and memory costs in deep learning for resource-constrained environments.
• To explore whether synaptic precision constraints can be leveraged to evolve more efficient, low-precision DNNs without sacrificing performance.
• To integrate synaptic precision constraints into a probabilistic generative evolutionary framework inspired by biological synaptogenesis.
• To evaluate the trade-off between model efficiency (synapse count, inference speed) and modeling accuracy (precision, recall) under precision constraints.

Proposed method

• Evolutionary synthesis uses a synaptic probability model $ P(S_g|W_{g-1}) $ as a genetic encoding, representing the 'DNA' of the network for heredity across generations.
• Offspring networks are stochastically synthesized via $ P(S_g) riangleq P(S_g|W_{g-1}) imes F(ar{ heta}) $, where $ F(ar{ heta}) $ represents environmental factors influencing network architecture.
• A probabilistic generative modeling strategy replaces traditional neuroevolution methods like genetic algorithms or evolutionary programming.
• After training at 32-bit precision, synaptic weights are constrained to 16-bit (half-precision) to enforce low-precision operation, influencing future generations.
• The process iteratively evolves networks over 13 generations, with inference speed, synapse count, and accuracy monitored on the Parse-27k dataset using Caffe and TensorRT.
• Environmental conditions are tuned to favor sparsity and efficiency, guiding the stochastic synaptogenesis process toward compact, high-speed architectures.

Experimental results

Research questions

• RQ1 How does imposing 16-bit synaptic precision constraints affect the evolutionary synthesis of deep neural networks in terms of model efficiency and performance?
• RQ2 Can synaptic precision reduction be effectively integrated into a probabilistic evolutionary framework to produce compact, high-speed DNNs?
• RQ3 To what extent is modeling accuracy preserved when synaptic precision is restricted to half-precision (16-bit) during evolution?
• RQ4 How does the number of synapses and inference speed evolve across generations under precision constraints?
• RQ5 Can the evolutionary synthesis process generate DNNs that are both computationally efficient and accurate in resource-constrained inference scenarios?

Key findings

• The 13th generation of evolved networks achieved an inference speed of 37 fps on the Jetson TX1, a 430% improvement from the first generation's 7 fps.
• The number of synapses decreased by a factor of ten over 13 generations, indicating significant architectural sparsification.
• Precision remained largely stable, increasing slightly by approximately 2% from the first to the 13th generation.
• Recall decreased marginally by about 2% across generations, indicating minimal degradation in detection sensitivity.
• The combination of synaptic precision restriction and evolutionary synthesis maintained high modeling performance while drastically reducing computational and memory demands.
• The integration of 16-bit precision constraints into the evolutionary process effectively guided the emergence of efficient, low-precision DNNs suitable for edge deployment.