[Paper Review] Dynamic Neural Networks: A Survey

Yizeng Han, Gao Huang+4 more|arXiv (Cornell University)|Feb 9, 2021

Advanced Neural Network Applications267 references40 citations

TL;DR

This survey organizes dynamic neural networks into sample-wise, spatial-wise, and temporal-wise categories, and reviews architecture design, decision strategies, training techniques, applications, and future directions.

ABSTRACT

Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc. In this survey, we comprehensively review this rapidly developing area by dividing dynamic networks into three main categories: 1) instance-wise dynamic models that process each instance with data-dependent architectures or parameters; 2) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data and 3) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts. The important research problems of dynamic networks, e.g., architecture design, decision making scheme, optimization technique and applications, are reviewed systematically. Finally, we discuss the open problems in this field together with interesting future research directions.

Motivation & Objective

• Provide an overview of dynamic neural networks and their advantages over static models.
• Categorize dynamic networks by granularity (sample-wise, spatial-wise, temporal-wise).
• Summarize architecture design, decision strategies, optimization techniques, and applications.
• Discuss open problems and future research directions in dynamic networks.

Proposed method

• Classify dynamic networks into three granularities: sample-wise, spatial-wise, and temporal-wise.
• Systematically review architecture design choices (depth, width, routing) and dynamic parameter mechanisms (adjustment, generation, and soft attention).
• Describe decision-making schemes (early exiting, gating, MoE, dynamic routing) and training considerations.
• Synthesize applications across vision and language tasks and relate CV and NLP developments.

Experimental results

Research questions

• RQ1 What are the main categories of dynamic neural networks and their defining characteristics?
• RQ2 What decision-making strategies and training techniques enable effective dynamic inference?
• RQ3 What are the common applications and practical considerations of dynamic networks across domains?
• RQ4 What open problems and future directions are identified for dynamic networks?

Key findings

• Dynamic networks can improve efficiency by adaptively allocating computation to easy samples or informative locations.
• Dynamic architectures and parameters expand representational power with modest cost increases.
• Adaptive inference enables better trade-offs between accuracy and computational budgets across platforms.
• There is strong compatibility with existing techniques (NAS, pruning, quantization, distillation) and broad applicability to vision and language tasks.
• Interpretability gains may arise from observing which components activate for given inputs.
• The surveyed approaches provide a unified view of diverse methods (early exits, MoE, dynamic routing, gating, and parameter prediction).