[Paper Review] Dynamically Throttleable Neural Networks (TNN)

Hengyue Liu, Samyak Parajuli+3 more|arXiv (Cornell University)|Nov 1, 2020

Advanced Neural Network Applications47 references4 citations

TL;DR

This paper introduces Dynamically Throttleable Neural Networks (TNN), a runtime-adaptive framework that enables deep neural networks to self-regulate performance and resource usage via a single utilization control parameter. By training TNN modules with runtime dropout and decoupling a context-aware controller, TNN achieves up to 74% latency reduction and 52% energy savings while maintaining peak accuracy on image and video classification tasks.

ABSTRACT

Conditional computation for Deep Neural Networks (DNNs) reduce overall computational load and improve model accuracy by running a subset of the network. In this work, we present a runtime throttleable neural network (TNN) that can adaptively self-regulate its own performance target and computing resources. We designed TNN with several properties that enable more flexibility for dynamic execution based on runtime context. TNNs are defined as throttleable modules gated with a separately trained controller that generates a single utilization control parameter. We validate our proposal on a number of experiments, including Convolution Neural Networks (CNNs such as VGG, ResNet, ResNeXt, DenseNet) using CiFAR-10 and ImageNet dataset, for object classification and recognition tasks. We also demonstrate the effectiveness of dynamic TNN execution on a 3D Convolustion Network (C3D) for a hand gesture task. Results show that TNN can maintain peak accuracy performance compared to vanilla solutions, while providing a graceful reduction in computational requirement, down to 74% reduction in latency and 52% energy savings.

Motivation & Objective

• Address the inefficiency of static deep neural networks in real-world environments with fluctuating resource constraints.
• Overcome the trade-off between model accuracy and computational efficiency by enabling runtime adaptation of network topology.
• Decouple the controller from the main network to allow flexible, application-specific throttling policies.
• Enable efficient, low-overhead conditional computation through a single scalar utilization parameter.
• Demonstrate practical viability of dynamic throttling in embedded systems with real power and latency measurements.

Proposed method

• Design TNN modules as trainable, filter-based blocks that can be conditionally activated or deactivated during inference.
• Use a single input-dependent utilization parameter $u$ to gate TNN modules, enabling flexible and low-complexity control.
• Train TNNs with runtime dropout at varying $u$ levels to regularize the network and ensure robustness across throttling points.
• Decouple the controller from the TNN architecture, allowing any policy—rule-based or learnable (e.g., via reinforcement learning)—to generate $u$.
• Implement a context-aware controller that selects optimal throttling levels based on input context, such as battery level or environmental conditions.
• Use a data-driven controller trained via reinforcement learning or contextual bandits to optimize $u$ for application-level performance metrics.

Experimental results

Research questions

• RQ1 Can a single, learnable utilization parameter enable effective and efficient dynamic throttling of deep neural networks across diverse inference workloads?
• RQ2 How does TNN maintain high accuracy while reducing computational load and energy consumption in real-world deployment scenarios?
• RQ3 To what extent can a decoupled, context-aware controller improve performance compared to fixed-throttling or naive gating strategies?
• RQ4 Can TNN be effectively applied to both image and video-based tasks, including those with temporal dynamics?
• RQ5 What are the practical gains in latency and energy efficiency when deploying TNN on embedded hardware with real-time measurements?

Key findings

• TNN maintains peak accuracy comparable to vanilla DNNs across all throttling levels, including at 74% lower latency and 52% energy savings on ImageNet and CIFAR-10.
• The TNN framework enables graceful performance degradation with increasing throttling, avoiding catastrophic accuracy loss even at high dropout levels.
• The context-aware controller trained via reinforcement learning outperforms both the vanilla architecture and all fixed-utilization settings in a 3D CNN for hand gesture recognition.
• Physical measurements on an embedded GPU confirm the practicality of TNN, demonstrating measurable reductions in runtime and power consumption.
• The single-parameter control mechanism enables seamless integration with diverse control policies, including heuristic rules and learnable controllers.
• TNN’s design allows for plug-and-play deployment of throttleable modules across various architectures, including VGG, ResNet, ResNeXt, and DenseNet.