[Paper Review] LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition

Ramyad Hadidi, Bahar Asgari+7 more|arXiv (Cornell University)|Mar 13, 2020

Advanced Neural Network Applications95 references4 citations

TL;DR

This paper proposes Low-Communication Parallelization (LCP), a method that splits deep neural networks into multiple narrow, independent branches to enable efficient, low-overhead distributed inference on edge devices. By minimizing inter-device communication and reducing per-node computation and memory, LCP achieves up to 56× speedup on edge hardware, with average speedups of 7×—improvable to 33× with pruning and quantization.

ABSTRACT

Deep neural networks (DNNs) have inspired new studies in myriad edge applications with robots, autonomous agents, and Internet-of-things (IoT) devices. However, performing inference of DNNs in the edge is still a severe challenge, mainly because of the contradiction between the intensive resource requirements of DNNs and the tight resource availability in several edge domains. Further, as communication is costly, taking advantage of other available edge devices by using data- or model-parallelism methods is not an effective solution. To benefit from available compute resources with low communication overhead, we propose the first DNN parallelization method for reducing the communication overhead in a distributed system. We propose a low-communication parallelization (LCP) method in which models consist of several almost-independent and narrow branches. LCP offers close-to-minimum communication overhead with better distribution and parallelization opportunities while significantly reducing memory footprint and computation compared to data- and model-parallelism methods. We deploy LCP models on three distributed systems: AWS instances, Raspberry Pis, and PYNQ boards. We also evaluate the performance of LCP models on a customized hardware (tailored for low latency) implemented on a small edge FPGA and as a 16mW 0.107mm2 ASIC @7nm chip. LCP models achieve a maximum and average speedups of 56x and 7x, compared to the originals, which could be improved by up to an average speedup of 33x by incorporating common optimizations such as pruning and quantization.

Motivation & Objective

• Address the challenge of performing efficient deep neural network (DNN) inference on resource-constrained edge devices such as IoTs, robots, and drones.
• Overcome the limitations of existing data- and model-parallelism methods, which suffer from high communication overhead and poor scalability in edge environments.
• Enable effective distribution of DNN inference across multiple edge devices with minimal communication, low memory footprint, and reduced computation per node.
• Design a method that is orthogonal to existing optimization techniques like pruning and quantization, enhancing their effectiveness in edge deployment.

Proposed method

• Split a single wide, deep DNN into multiple narrow, independent branches that process different feature representations in parallel.
• Restrict communication to only input data and pre-final layer activations, drastically reducing inter-device data transfer compared to traditional model parallelism.
• Ensure each branch maintains near-identical accuracy to the original model by slightly fattening the branches after splitting, while still reducing total parameters.
• Use a memory layout preprocessing step that reorders weights and activations into sequential, streaming-access patterns to overlap computation and data transfer.
• Implement the LCP architecture on diverse platforms including AWS, Raspberry Pi, PYNQ FPGAs, and a custom 7nm ASIC, demonstrating portability and efficiency.
• Leverage systolic array hardware accelerators on FPGA and ASIC to further optimize latency and energy efficiency for edge inference.

Experimental results

Research questions

• RQ1 Can a DNN inference method be designed that minimizes communication overhead while enabling effective parallelization across multiple edge devices?
• RQ2 How does the proposed LCP method compare to data- and model-parallelism in terms of communication, memory, and computation overhead on edge platforms?
• RQ3 To what extent can LCP reduce memory and computation footprint per device while maintaining or recovering model accuracy?
• RQ4 Can LCP be effectively deployed across heterogeneous edge systems, including low-power FPGAs and ASICs, with measurable performance gains?
• RQ5 How does combining LCP with existing optimization techniques like pruning and quantization affect inference speedup and energy efficiency?

Key findings

• LCP achieves a maximum speedup of 56× and an average speedup of 7× over the original DNN on edge platforms including Raspberry Pi, AWS, and PYNQ boards.
• When combined with pruning and quantization, the average speedup of LCP models improves to 33×, demonstrating strong synergy with existing model compression techniques.
• LCP reduces the total number of parameters in the model after re-optimization (fattening) compared to the original network, despite slight architectural expansion.
• The method enables inter-layer parallelism due to the independence of branches, unlike model parallelism which is limited to intra-layer parallelism due to sequential dependencies.
• On a custom 7nm ASIC, LCP achieves 16mW power consumption and a chip area of only 0.107mm², demonstrating high energy and area efficiency.
• The LCP design supports efficient memory access through sequential data layout, enabling overlapping of data transfers and computation, which mitigates memory wall effects.