[Paper Review] Designing Network Design Strategies Through Gradient Path Analysis

Chien-Yao Wang, Hong-Yuan Mark Liao+1 more|arXiv (Cornell University)|Nov 9, 2022

Machine Learning and ELM179 citations

TL;DR

The paper advocates gradient-path based network design, proposing layer-, stage-, and network-level strategies (PRN, CSPNet, ELAN) to maximize gradient diversity and improve learning efficiency.

ABSTRACT

Designing a high-efficiency and high-quality expressive network architecture has always been the most important research topic in the field of deep learning. Most of today's network design strategies focus on how to integrate features extracted from different layers, and how to design computing units to effectively extract these features, thereby enhancing the expressiveness of the network. This paper proposes a new network design strategy, i.e., to design the network architecture based on gradient path analysis. On the whole, most of today's mainstream network design strategies are based on feed forward path, that is, the network architecture is designed based on the data path. In this paper, we hope to enhance the expressive ability of the trained model by improving the network learning ability. Due to the mechanism driving the network parameter learning is the backward propagation algorithm, we design network design strategies based on back propagation path. We propose the gradient path design strategies for the layer-level, the stage-level, and the network-level, and the design strategies are proved to be superior and feasible from theoretical analysis and experiments.

Motivation & Objective

• Redefine network design strategies by analyzing how objective functions influence gradient flow during backpropagation.
• Propose layer-, stage-, and network-level architectures that maximize gradient combinations and propagation efficiency.
• Demonstrate theoretical and empirical advantages of gradient-path design over traditional data-path design.

Proposed method

• Introduce gradient path design strategies that analyze backward gradients rather than solely forward feature paths.
• Develop Partial Residual Network (PRN) with masked and asymmetric residual layers to increase gradient combinations.
• Design Cross Stage Partial Network (CSPNet) to maximize gradient sources and timestamps while improving hardware efficiency.
• Propose Efficient Layer Aggregation Network (ELAN) to optimize gradient propagation paths during network scaling.
• Analyze gradient combinations via gradient timestamps and gradient sources to explain performance gains.
• Compare gradient-path designs against data-path baselines through ablations on MS COCO with YOLOR-v3/YOLO-v5 baselines.

Experimental results

Research questions

• RQ1 How does gradient propagation path influence learning efficiency and parameter utilization in CNNs?
• RQ2 Can layer-, stage-, and network-level gradient-path designs yield measurable gains in accuracy and speed over traditional data-path designs?
• RQ3 What are the effects of gradient timestamps and gradient sources on network performance across architectures?
• RQ4 How do PRN, CSPNet, and ELAN perform on object detection and instance segmentation benchmarks?

Key findings

• PRN improves object detection AP by 0.5 and instance segmentation AP by 0.1/0.3 on COCO compared to YOLOR-v3.
• CSPNet reduces FLOPs by up to 30% and can improve AP (e.g., 1.5% on object detection) while maintaining or improving speed.
• ELAN achieves up to 0.7% AP gain with 35% less computation in YOLOR-ELAN configurations.
• Stop-gradient ablations show shortening gradient paths can improve convergence in deep networks.
• Experiments across PRN, CSPNet, and ELAN demonstrate consistent gains over baseline architectures on COCO.