[Paper Review] Deep Learning-Based Video Coding: A Review and A Case Study

Dong Liu, Yue Li+3 more|arXiv (Cornell University)|Apr 29, 2019

Advanced Image Processing Techniques94 references67 citations

TL;DR

The paper reviews deep learning approaches for image/video coding, categorizes them into deep schemes and deep tools, and presents a DLVC prototype with CNN-based in-loop filtering and CNN-BARC showing substantial bitrate savings over HEVC.

ABSTRACT

The past decade has witnessed great success of deep learning technology in many disciplines, especially in computer vision and image processing. However, deep learning-based video coding remains in its infancy. This paper reviews the representative works about using deep learning for image/video coding, which has been an actively developing research area since the year of 2015. We divide the related works into two categories: new coding schemes that are built primarily upon deep networks (deep schemes), and deep network-based coding tools (deep tools) that shall be used within traditional coding schemes or together with traditional coding tools. For deep schemes, pixel probability modeling and auto-encoder are the two approaches, that can be viewed as predictive coding scheme and transform coding scheme, respectively. For deep tools, there have been several proposed techniques using deep learning to perform intra-picture prediction, inter-picture prediction, cross-channel prediction, probability distribution prediction, transform, post- or in-loop filtering, down- and up-sampling, as well as encoding optimizations. In the hope of advocating the research of deep learning-based video coding, we present a case study of our developed prototype video codec, namely Deep Learning Video Coding (DLVC). DLVC features two deep tools that are both based on convolutional neural network (CNN), namely CNN-based in-loop filter (CNN-ILF) and CNN-based block adaptive resolution coding (CNN-BARC). Both tools help improve the compression efficiency by a significant margin. With the two deep tools as well as other non-deep coding tools, DLVC is able to achieve on average 39.6\% and 33.0\% bits saving than HEVC, under random-access and low-delay configurations, respectively. The source code of DLVC has been released for future researches.

Motivation & Objective

• Survey the state of deep learning for image/video coding up to 2018.
• Distinguish between deep schemes and deep tools and analyze their pros/cons.
• Demonstrate a practical Deep Learning Video Coding (DLVC) prototype to illustrate gains.

Proposed method

• Review representative deep coding works using pixel probability modeling and auto-encoder approaches.
• Classify deep schemes as predictive/pixel-probability models or transform-based auto-encoders, and discuss deep tools used within traditional codecs.
• Present a case study DLVC with two CNN-based tools (CNN-ILF and CNN-BARC) and report compression gains.
• Provide experimental comparison to HEVC across configurations to illustrate potential gains and tradeoffs.

Experimental results

Research questions

• RQ1 What are the main deep learning paradigms applied to image/video coding and how do they differ in approach and objective?
• RQ2 What is the comparative performance and practicality of deep schemes versus deep tools in achieving compression gains?
• RQ3 How do CNN-based tools within a traditional codec affect rate-distortion performance and complexity in DLVC?

Key findings

• Deep schemes can match or exceed traditional image coding efficiency in some cases, but video coding gains remain more modest relative to HEVC so far.
• Deep tools embedded in traditional codecs show promising compression beyond HEVC in some aspects.
• The DLVC prototype with CNN-ILF and CNN-BARC achieves average bitrate savings of 39.6% (random-access) and 33.0% (low-delay) over HEVC.
• The paper emphasizes tradeoffs among compression efficiency, encoding/decoding complexity, perceptual quality, universal applicability, and tool integration.
• Code for DLVC has been released for future research.