[Paper Review] Layer Folding: Neural Network Depth Reduction using Activation Linearization

Amir Ben Dror, Niv Zehngut+4 more|arXiv (Cornell University)|Jun 17, 2021

Advanced Neural Network Applications63 references4 citations

TL;DR

This paper proposes Layer Folding, a method to reduce neural network depth by learning to remove non-linear activations between consecutive convolutional layers, enabling the merging of those layers into a single larger convolution. The approach achieves significant latency reduction—up to 25%—on mobile and edge devices with minimal accuracy drop (as low as 0.5%) while maintaining or improving performance on ImageNet benchmarks.

ABSTRACT

Despite the increasing prevalence of deep neural networks, their applicability in resource-constrained devices is limited due to their computational load. While modern devices exhibit a high level of parallelism, real-time latency is still highly dependent on networks' depth. Although recent works show that below a certain depth, the width of shallower networks must grow exponentially, we presume that neural networks typically exceed this minimal depth to accelerate convergence and incrementally increase accuracy. This motivates us to transform pre-trained deep networks that already exploit such advantages into shallower forms. We propose a method that learns whether non-linear activations can be removed, allowing to fold consecutive linear layers into one. We apply our method to networks pre-trained on CIFAR-10 and CIFAR-100 and find that they can all be transformed into shallower forms that share a similar depth. Finally, we use our method to provide more efficient alternatives to MobileNetV2 and EfficientNet-Lite architectures on the ImageNet classification task.

Motivation & Objective

• Address the challenge of high latency in deep neural networks on resource-constrained devices by reducing network depth without significant accuracy degradation.
• Investigate whether pre-trained deep networks can be transformed into shallower, more efficient forms while preserving their representational power.
• Identify a minimal depth threshold—termed Effective Degree of Non-Linearity (EDNL)—beyond which networks can be compressed without losing performance.
• Develop a method that enables depth reduction by learning to remove ReLU-like activations between layers, allowing functional merging of adjacent linear layers.
• Provide efficient alternatives to popular mobile architectures like MobileNetV2 and EfficientNet-Lite through depth reduction via activation linearization.

Proposed method

• Propose a technique to learn which non-linear ReLU6 activations can be removed between consecutive convolutional layers without degrading model accuracy.
• Formally define the transformation as folding consecutive linear layers into a single equivalent layer by removing intermediate non-linearities, resulting in a larger kernel size: from $k \times k$ to $(2k-1) \times (2k-1)$.
• Apply the method during a fine-tuning phase that preserves the original network's intermediate representations, avoiding the need to re-train from scratch.
• Use a shared learnable parameter $\alpha$ across paired ReLU6 activations to ensure they are either both removed or both retained, enabling consistent folding per block.
• Apply the method to entire MBConv blocks in MobileNetV2 and EfficientNet-Lite, folding expansion, depthwise, and projection layers into a single convolution when both ReLU6 activations are removed.
• Leverage hardware-aware latency optimization by prioritizing reductions in inter-layer computational overhead, especially beneficial for accelerators and GPUs with high intra-layer parallelism.

Experimental results

Research questions

• RQ1 Can pre-trained deep neural networks be transformed into shallower forms with minimal accuracy degradation by removing non-linear activations?
• RQ2 What is the minimal depth (Effective Degree of Non-Linearity, EDNL) required to preserve performance on a given task, and is it consistent across different architectures?
• RQ3 Does removing intermediate ReLU6 activations between consecutive convolutions enable functional merging of layers while reducing latency on edge devices?
• RQ4 How does the proposed Layer Folding method compare to existing pruning and re-parameterization techniques in terms of efficiency and accuracy trade-offs?
• RQ5 Can the method be applied effectively to state-of-the-art mobile architectures like MobileNetV2 and EfficientNet-Lite to yield faster inference with minimal FLOPs increase?

Key findings

• Layer Folding reduces inference latency by up to 25% on ImageNet classification tasks, with the most significant gains observed on MobileNetV2-1.0 and MobileNetV2-1.4.
• The method achieves a 1.2% higher top-1 accuracy than MobileNetV2-0.75 while being 14% faster, demonstrating improved efficiency-accuracy trade-offs.
• On MobileNetV2-1.4, the approach reduces latency by 19% and FLOPs by 3%, with only a 0.5% accuracy drop, showing strong practical viability.
• For EfficientNet-Lite0, latency is reduced by 15% with a 3% FLOPs reduction and only 0.5% accuracy drop, indicating broad applicability across architectures.
• Folding entire MBConv blocks by removing both ReLU6 activations results in halving the computational load for blocks with expansion factor $t=6$, due to merging three convolutions into one.
• The method enables a favorable trade-off: although FLOPs may increase slightly in some cases (e.g., when $c \gg 9$), the reduction in inter-layer overhead leads to net latency gains on hardware accelerators.