[论文解读] Deep 3D Pan via Local adaptive "t-shaped" convolutions with global and local adaptive dilations

Juan Luis Gonzalez Bello, Munchurl Kim|arXiv (Cornell University)|Apr 30, 2020

Advanced Vision and Imaging参考文献 39被引用 3

一句话总结

本文提出Monster-Net，一种使用T形自适应卷积与全局及局部自适应空洞率的深度学习架构，以实现高质量的单图像3D全景（Deep 3D Pan）合成。该方法通过有效建模来自单张2D输入图像的全局相机位移和局部3D几何结构，在视图合成与无监督单目深度估计任务中达到最先进性能。

ABSTRACT

Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereoscopic view synthesis at arbitrary camera positions along the X-axis, or Deep 3D Pan, with adaptive kernels equipped with globally and locally adaptive dilations. Our proposed network architecture, the monster-net, is devised with a novel t-shaped adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image's pixels for the synthesis of naturally looking 3D panned views when a 2-D input image is given. Extensive experiments were performed on the KITTI, CityScapes and our VXXLXX_STEREO indoors dataset to prove the efficacy of our method. Our monster-net significantly outperforms the state-of-the-art method, SOTA, by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity information that can be extracted from the kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirming the effectiveness of our method.

研究动机与目标

• 解决从单张2D输入图像合成逼真3D全景视图的挑战。
• 提升在单图像视图合成中对全局相机位移与局部3D几何结构的建模能力。
• 增强无监督单目深度预测中视差估计的可靠性。
• 在视图合成质量与几何一致性方面超越现有最先进方法。

提出的方法

• 该方法引入一种新颖的T形自适应核，通过同时利用全局与局部自适应空洞率动态调整感受野。
• 全局自适应空洞率将沿X轴的总体相机位移信息融入网络的特征学习过程。
• 局部自适应空洞率实现对目标视图中每个像素周围局部3D几何结构的细粒度建模。
• 网络架构命名为Monster-Net，整合这些自适应卷积操作，以在任意相机位置合成高保真立体视图。
• 自适应空洞机制在训练过程中端到端学习，使网络能够根据输入内容自动校准核扩展。
• 该方法复用相同的特征图生成合成视图与视差图，从而提升一致性与可靠性。

实验结果

研究问题

• RQ1 自适应空洞率能否提升从单张2D图像合成的3D全景视图的质量与几何一致性？
• RQ2 将全局相机位移与局部3D几何结构相结合，对视图合成性能有何影响？
• RQ3 所提出的网络能否在无监督单目深度估计任务中超越最先进方法，取得更优结果？
• RQ4 T形自适应核在多大程度上增强了视图合成任务的特征表示能力？

主要发现

• Monster-Net在KITTI、CityScapes与VXXLXX_STEREO数据集上的所有指标（包括RMSE、PSNR与SSIM）均显著优于最先进方法。
• 与基线方法相比，合成图像展现出更可靠的图像结构与更一致的3D几何形态。
• Monster-Net预测的视差图比最先进方法更准确、更一致，证实了其在深度估计能力上的提升。
• 该方法在多种场景（包括室内与室外环境）中均表现出强大的泛化能力，已在多个基准数据集上得到验证。