[论文解读] Self-Supervised Learning for Stereo Matching with Self-Improving Ability

Yiran Zhong, Yuchao Dai+1 more|arXiv (Cornell University)|Sep 4, 2017

Advanced Vision and Imaging参考文献 9被引用 154

一句话总结

该论文提出了一种自监督的端到端立体匹配网络，通过立体图像对在没有地面实测地图的情况下学习密集视差，利用图像扭曲损失和回环一致性，并在新环境中进行在线自我改进。

ABSTRACT

Exiting deep-learning based dense stereo matching methods often rely on ground-truth disparity maps as the training signals, which are however not always available in many situations. In this paper, we design a simple convolutional neural network architecture that is able to learn to compute dense disparity maps directly from the stereo inputs. Training is performed in an end-to-end fashion without the need of ground-truth disparity maps. The idea is to use image warping error (instead of disparity-map residuals) as the loss function to drive the learning process, aiming to find a depth-map that minimizes the warping error. While this is a simple concept well-known in stereo matching, to make it work in a deep-learning framework, many non-trivial challenges must be overcome, and in this work we provide effective solutions. Our network is self-adaptive to different unseen imageries as well as to different camera settings. Experiments on KITTI and Middlebury stereo benchmark datasets show that our method outperforms many state-of-the-art stereo matching methods with a margin, and at the same time significantly faster.

研究动机与目标

• 解决立体匹配中对地面实测视差地图的依赖。
• 提出一个端到端网络，在没有监督的情况下从立体输入学习密集视差。
• 引入基于图像扭曲的损失和3D特征正则化以避免简单解。
• 通过在线适应未见过的成像和相机设置来实现自我改进。

提出的方法

• 使用残差18层3x3卷积网络提取局部特征，输出64通道特征。
• 通过在视差范围内连接左右特征来构建跨视角特征体。
• 使用残差连接的自顶向下模块（Res-TDM）进行3D特征匹配，生成3D视差体积。
• 通过soft-argmin操作将3D体积投影为2D视差图。
• 评估左右图像之间的扭曲质量，并以光度、梯度和基于SSIM的损失作为监督。
• 结合回环一致性损失和最大深度启发式以处理纹理缺失区域并强制对称约束。

实验结果

研究问题

• RQ1 是否可以在不使用地面真实深度图的情况下通过自监督学习得到密集的立体视差？
• RQ2 图像重建（扭曲）误差是否足以监督端到端的立体学习？
• RQ3 如何在3D特征空间中的正则化和回环一致性提高纹理缺失区域的性能？
• RQ4 模型是否能够在无标注数据的情况下对未见场景和相机设置进行在线适应？

主要发现

• 该自监督网络在KITTI和Middlebury基准测试上在没有地面真实视差的情况下取得具有竞争力的性能。
• 模型在接触到新的立体数据时表现出自我改进（在线适应）的能力。
• 推理时间对立体对来说是实用的，可选的在线微调会增加运行时。
• 回环一致性和3D正则化有助于缓解纹理缺失或具有挑战性区域的退化解。
• 该方法从在KITTI上训练的模型适应到Middlebury，在在线迭代后取得显著改进。