[Paper Review] MobileCodec: Neural Inter-frame Video Compression on Mobile Devices

Hoang Le, Liang Zhang+7 more|arXiv (Cornell University)|Jul 18, 2022

CCD and CMOS Imaging Sensors4 citations

TL;DR

MobileCodec presents the first real-time neural inter-frame video decoder for mobile devices, leveraging Qualcomm's AI Engine to achieve 31 FPS decoding of 720p HD videos at low bitrate with high visual quality. The system uses an efficient, asymmetric architecture with quantization-aware training and parallel entropy coding to enable practical deployment of neural video compression on power-constrained mobile hardware.

ABSTRACT

Realizing the potential of neural video codecs on mobile devices is a big technological challenge due to the computational complexity of deep networks and the power-constrained mobile hardware. We demonstrate practical feasibility by leveraging Qualcomm's technology and innovation, bridging the gap from neural network-based codec simulations running on wall-powered workstations, to real-time operation on a mobile device powered by Snapdragon technology. We show the first-ever inter-frame neural video decoder running on a commercial mobile phone, decoding high-definition videos in real-time while maintaining a low bitrate and high visual quality.

Motivation & Objective

• To demonstrate the feasibility of real-time neural video decoding on commercial mobile hardware, overcoming computational and power constraints.
• To design an efficient, asymmetric neural video codec architecture tailored for mobile deployment with low latency and high energy efficiency.
• To develop a parallel entropy coding algorithm compatible with neural entropy models for efficient bitstream generation.
• To enable practical deployment of end-to-end learned video compression on mobile platforms, bridging the gap between simulation and real-world use.
• To achieve competitive rate-distortion performance while maintaining real-time decoding speed on mobile devices using optimized inference on the Qualcomm AI Engine.

Proposed method

• Design an asymmetric neural video codec with separate, lightweight networks for encoder and decoder, reducing computational load on the receiver.
• Implement joint motion compensation and residual coding in feature space to improve compression efficiency and reduce redundancy.
• Apply quantization-aware training to convert full-precision neural networks to 8-bit fixed-point inference, enabling efficient execution on mobile AI accelerators.
• Develop a parallel entropy coding algorithm that efficiently encodes outputs from neural entropy models, reducing encoding latency and improving throughput.
• Optimize network architecture by reducing filter counts at high resolutions and using smaller models on the receiver side to minimize MAC operations.
• Leverage the Qualcomm AI Engine for hardware-accelerated inference, ensuring real-time performance on commercial mobile devices.

Experimental results

Research questions

• RQ1 Can a neural inter-frame video codec achieve real-time decoding on a commercial mobile device under power and latency constraints?
• RQ2 How can neural video compression be optimized for mobile deployment without sacrificing rate-distortion performance or visual quality?
• RQ3 What architectural and algorithmic modifications are necessary to make neural codecs practical on mobile hardware with limited computational resources?
• RQ4 Can parallel entropy coding be effectively integrated with neural entropy models to improve encoding efficiency on mobile platforms?
• RQ5 What is the achievable trade-off between bitrate, visual quality, and decoding speed in a mobile-optimized neural video codec?

Key findings

• MobileCodec achieves real-time decoding at 31 frames per second (FPS) on a commercial mobile device, demonstrating the first public real-time neural inter-frame video decoder on mobile hardware.
• The system decodes 720p HD videos with an average PSNR of 40.18 dB and an average MSSSIM of 0.9890 across diverse video sequences from the Pexels dataset.
• The receiver-side computation is up to two times lower than the sender-side, with a total of 390.1 KMACs per pixel required for decoding a single P-frame.
• The model achieves competitive performance compared to fast H.264 and H.265 encodings in ffmpeg, particularly at low bitrates, with superior visual quality at similar bitrates.
• The total parameter count of the decoder is 11.589 million, with 390.1 KMACs per pixel, indicating strong efficiency for mobile deployment.
• The system demonstrates that neural video codecs can be successfully deployed on mobile platforms through model quantization, architectural optimization, and hardware-aware design.