[Paper Review] Compressing Representations for Embedded Deep Learning

Juliano S. Assine, Alan Godoy+1 more|arXiv (Cornell University)|Nov 23, 2019

IoT and Edge/Fog Computing12 references4 citations

TL;DR

This paper proposes a distributed deep learning inference framework that splits MobileNetV2 across edge and cloud by compressing intermediate feature representations, using PCA-based compression to minimize communication costs. It identifies a 'sweet spot' in the network's middle layers where compression maximizes accuracy while minimizing bandwidth, validated by theoretical alignment with information bottleneck principles and empirical results showing significant bandwidth reduction over JPEG-compressed inputs.

ABSTRACT

Despite recent advances in architectures for mobile devices, deep learning computational requirements remains prohibitive for most embedded devices. To address that issue, we envision sharing the computational costs of inference between local devices and the cloud, taking advantage of the compression performed by the first layers of the networks to reduce communication costs. Inference in such distributed setting would allow new applications, but requires balancing a triple trade-off between computation cost, communication bandwidth, and model accuracy. We explore that trade-off by studying the compressibility of representations at different stages of MobileNetV2, showing those results agree with theoretical intuitions about deep learning, and that an optimal splitting layer for network can be found with a simple PCA-based compression scheme.

Motivation & Objective

• To address the challenge of deploying deep learning on embedded systems with limited computation, memory, and network bandwidth.
• To enable feasible real-time computer vision applications in IoT by offloading part of the inference to the cloud.
• To minimize the trade-off between computational cost on the device, network bandwidth usage, and model accuracy in distributed inference.
• To identify an optimal model-splitting point (layer k) that balances compression efficiency and predictive performance.
• To validate theoretical intuitions from information theory—particularly the information bottleneck—on real-world deep learning representations.

Proposed method

• The method splits MobileNetV2 inference into two stages: local processing on the edge device up to layer k, followed by cloud-based completion.
• Intermediate feature maps at layer k are compressed using blocked principal component analysis (PCA), followed by quantization and Huffman coding.
• The compression scheme is applied across different layers of MobileNetV2 to evaluate trade-offs between bandwidth, accuracy, and computation.
• The approach leverages theoretical insights from the information bottleneck, where representations in deeper layers are already compressed and less amenable to further compression.
• The model is evaluated on ImageNet using the highest-accuracy MobileNetV2 configuration (α=1.0, input size=224) with varying split points k.
• Bandwidth costs are measured relative to transmitting JPEG-compressed images, establishing a baseline for comparison.

Experimental results

Research questions

• RQ1 Where in a deep neural network is the optimal trade-off between representation compressibility and predictive accuracy for distributed inference?
• RQ2 Can PCA-based compression of intermediate features significantly reduce communication costs without degrading model accuracy?
• RQ3 How do theoretical principles like the information bottleneck and linearization of deep representations align with empirical results in compressed feature representations?
• RQ4 Does compressing intermediate features enable new embedded deep learning applications previously infeasible due to bandwidth or computational constraints?
• RQ5 What is the impact of model architecture and input representation on the effectiveness of intermediate feature compression in edge-cloud inference?

Key findings

• A clear 'sweet spot' for model splitting was found in the middle layers of MobileNetV2, where compression yields the best balance of bandwidth reduction and accuracy preservation.
• PCA-based compression of intermediate features reduced communication costs to less than 10% of the bandwidth required to transmit JPEG-compressed images, despite using a simple compression scheme.
• Earlier layers in the network were poorly compressible with PCA, requiring more complex methods like JPEG, while later layers were already highly compressed and sensitive to further compression.
• The results strongly aligned with the information bottleneck theory, showing that mutual information between input and representation decreases with depth, even as data structure grows.
• The computational overhead of compression on the edge device was negligible compared to the cost of deep learning inference, making it feasible for real-world deployment.
• The study demonstrates that combining theoretical insights with practical compression techniques enables efficient, scalable, and accurate distributed inference for embedded and IoT systems.