[论文解读] Deep Learning-Based Video Coding: A Review and A Case Study

Dong Liu, Yue Li+3 more|arXiv (Cornell University)|Apr 29, 2019

Advanced Image Processing Techniques参考文献 94被引用 67

一句话总结

这篇论文回顾了用于图像/视频编码的深度学习方法，将它们分为深度方案和深度工具，并展示了一个 DLVC 原型，包含基于 CNN 的环路滤波和 CNN-BARC，在比特率方面优于 HEVC 的显著节省。

ABSTRACT

The past decade has witnessed great success of deep learning technology in many disciplines, especially in computer vision and image processing. However, deep learning-based video coding remains in its infancy. This paper reviews the representative works about using deep learning for image/video coding, which has been an actively developing research area since the year of 2015. We divide the related works into two categories: new coding schemes that are built primarily upon deep networks (deep schemes), and deep network-based coding tools (deep tools) that shall be used within traditional coding schemes or together with traditional coding tools. For deep schemes, pixel probability modeling and auto-encoder are the two approaches, that can be viewed as predictive coding scheme and transform coding scheme, respectively. For deep tools, there have been several proposed techniques using deep learning to perform intra-picture prediction, inter-picture prediction, cross-channel prediction, probability distribution prediction, transform, post- or in-loop filtering, down- and up-sampling, as well as encoding optimizations. In the hope of advocating the research of deep learning-based video coding, we present a case study of our developed prototype video codec, namely Deep Learning Video Coding (DLVC). DLVC features two deep tools that are both based on convolutional neural network (CNN), namely CNN-based in-loop filter (CNN-ILF) and CNN-based block adaptive resolution coding (CNN-BARC). Both tools help improve the compression efficiency by a significant margin. With the two deep tools as well as other non-deep coding tools, DLVC is able to achieve on average 39.6\% and 33.0\% bits saving than HEVC, under random-access and low-delay configurations, respectively. The source code of DLVC has been released for future researches.

研究动机与目标

• 对截至 2018 年的图像/视频编码深度学习现状进行综述。
• 区分深度方案与深度工具，并分析它们的优缺点。
• 演示一个实用的深度学习视频编码（DLVC）原型以说明收益。

提出的方法

• 使用像素概率建模和自编码器方法来回顾具有代表性的深度编码工作。
• 将深度方案分类为预测/像素概率模型或基于变换的自编码器，并讨论在传统编解码器中使用的深度工具。
• 给出一个包含两个基于 CNN 的工具（CNN-ILF 和 CNN-BARC）的 DLVC 案例研究，并报告压缩增益。
• 在不同配置下提供与 HEVC 的实验对比，以说明潜在收益和权衡。

实验结果

研究问题

• RQ1 应用于图像/视频编码的主要深度学习范式是什么，它们在方法和目标上有何不同？
• RQ2 深度方案与深度工具在实现压缩增益方面的性能与实用性有何差异？
• RQ3 在传统编解码器中基于 CNN 的工具如何影响 DLVC 的码率-失真性能和复杂度？

主要发现

• 深度方案在某些情况下可以达到甚至超过传统图像编码效率，但相对于 HEVC，视频编码方面的增益仍然较为温和。
• 嵌入传统编解码器的深度工具在某些方面展示出超越 HEVC 的有希望的压缩性能。
• 包含 CNN-ILF 和 CNN-BARC 的 DLVC 原型在随机访问模式下平均节省比特率为 39.6%，在低延迟模式下为 33.0%（相对于 HEVC）。
• 论文强调在压缩效率、编码/解码复杂度、感知质量、普遍适用性以及工具集成之间的权衡。
• DLVC 的代码已发布以供未来研究使用。