[Paper Review] Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms

Wei Lou, Xun Lei+4 more|arXiv (Cornell University)|May 8, 2021

Advanced Neural Network Applications24 references4 citations

TL;DR

Dynamic-OFA enables real-time DNN architecture switching on heterogeneous embedded platforms by leveraging a pre-trained Once-for-All (OFA) super-network to dynamically select optimal sub-networks at runtime without retraining. It achieves up to 3.5x faster inference (CPU) and 2.4x (GPU) for similar accuracy, or 3.8% (CPU) and 5.1% (GPU) higher accuracy at similar latency on Jetson Xavier NX.

ABSTRACT

Mobile and embedded platforms are increasingly required to efficiently execute computationally demanding DNNs across heterogeneous processing elements. At runtime, the available hardware resources to DNNs can vary considerably due to other concurrently running applications. The performance requirements of the applications could also change under different scenarios. To achieve the desired performance, dynamic DNNs have been proposed in which the number of channels/layers can be scaled in real time to meet different requirements under varying resource constraints. However, the training process of such dynamic DNNs can be costly, since platform-aware models of different deployment scenarios must be retrained to become dynamic. This paper proposes Dynamic-OFA, a novel dynamic DNN approach for state-of-the-art platform-aware NAS models (i.e. Once-for-all network (OFA)). Dynamic-OFA pre-samples a family of sub-networks from a static OFA backbone model, and contains a runtime manager to choose different sub-networks under different runtime environments. As such, Dynamic-OFA does not need the traditional dynamic DNN training pipeline. Compared to the state-of-the-art, our experimental results using ImageNet on a Jetson Xavier NX show that the approach is up to 3.5x (CPU), 2.4x (GPU) faster for similar ImageNet Top-1 accuracy, or 3.8% (CPU), 5.1% (GPU) higher accuracy at similar latency.

Motivation & Objective

• Address the challenge of dynamic performance degradation in DNN inference on heterogeneous embedded platforms due to fluctuating hardware resources and application workloads.
• Overcome the high training cost of traditional dynamic DNNs that require retraining for each deployment scenario.
• Enable efficient runtime adaptation across diverse computing units (CPU, GPU, NPU) using a single shared backbone model.
• Improve accuracy-latency trade-offs in real-time by selecting optimal sub-networks based on current hardware availability and application constraints.
• Eliminate the need for retraining by pre-sampling and pre-evaluating sub-networks from an OFA super-network offline.

Proposed method

• Pre-sample a family of sub-networks from a pre-trained Once-for-All (OFA) super-network, including variations in width, depth, filter size, and input resolution.
• Offline, evaluate all sub-networks on both CPU and GPU to build a Pareto-optimal look-up table of accuracy-latency trade-offs.
• Store pre-calculated batch normalization statistics to enable fast inference and sub-network switching.
• Deploy a runtime manager (RTM) that monitors real-time latency and accuracy constraints using a sliding window over 50 inferences.
• Dynamically switch between sub-networks at runtime based on available hardware resources and application requirements, minimizing switching overhead.
• Support coexistence of multiple Dynamic-OFA models on the same GPU by collectively adapting sub-networks to meet individual latency constraints.

Experimental results

Research questions

• RQ1 Can a pre-trained OFA super-network be leveraged to enable dynamic DNN architecture switching without additional training?
• RQ2 How effective is Dynamic-OFA in maintaining high accuracy while adapting to changing hardware resource availability on heterogeneous SoCs?
• RQ3 Can Dynamic-OFA achieve better accuracy-latency trade-offs than state-of-the-art dynamic DNNs on CPU and GPU workloads?
• RQ4 How does Dynamic-OFA manage performance constraints when multiple DNNs or concurrent workloads share the same GPU?
• RQ5 What is the runtime overhead of sub-network switching, and how does it impact real-time responsiveness?

Key findings

• Dynamic-OFA achieves up to 3.5x faster inference on CPU and 2.4x on GPU compared to state-of-the-art methods for similar ImageNet Top-1 accuracy.
• At similar latency, Dynamic-OFA delivers 3.8% higher Top-1 accuracy on CPU and 5.1% higher on GPU compared to baseline methods.
• The runtime manager successfully adapts sub-networks in response to changing latency constraints, switching from level 6 to level 2 on GPU to meet a stricter 40ms target.
• When GPU resources are shared with a concurrent training task, Dynamic-OFA reduces its sub-network level from 4 to 2, trading 2.6% accuracy to meet latency constraints.
• Two Dynamic-OFA models coexisting on the same GPU can collectively adapt their sub-networks to meet individual latency constraints, with model A switching from level 6 to 5 and model B from level 5 to 4.
• The runtime manager introduces only ~15ms overhead per switch, with reaction time of 1–2 seconds (approx. 50 inferences), making it practical for real-world deployment.