[论文解读] Self-Supervised Monocular Depth Estimation with Self-Reference Distillation and Disparity Offset Refinement

Zhong Liu, Ran Li+3 more|arXiv (Cornell University)|Feb 20, 2023

Advanced Vision and Imaging被引用 4

一句话总结

该论文提出一种基于自参考蒸馏与视差偏移优化的自监督单目深度估计方法。通过持续更新学生-教师模型对并利用多视角一致性过滤异常值，该方法提升了深度估计精度，减轻了上采样带来的错位问题，在KITTI和Make3D基准上实现了最先进性能。

ABSTRACT

Monocular depth estimation plays a fundamental role in computer vision. Due to the costly acquisition of depth ground truth, self-supervised methods that leverage adjacent frames to establish a supervisory signal have emerged as the most promising paradigms. In this work, we propose two novel ideas to improve self-supervised monocular depth estimation: 1) self-reference distillation and 2) disparity offset refinement. Specifically, we use a parameter-optimized model as the teacher updated as the training epochs to provide additional supervision during the training process. The teacher model has the same structure as the student model, with weights inherited from the historical student model. In addition, a multiview check is introduced to filter out the outliers produced by the teacher model. Furthermore, we leverage the contextual consistency between high-scale and low-scale features to obtain multiscale disparity offsets, which are used to refine the disparity output incrementally by aligning disparity information at different scales. The experimental results on the KITTI and Make3D datasets show that our method outperforms previous state-of-the-art competitors.

研究动机与目标

• 通过推进自监督单目深度估计，解决真实世界深度标注数据有限的挑战。
• 提升在动态或低纹理区域的训练稳定性与深度估计精度，这些区域是标准自监督方法失效的区域。
• 通过引入基于上下文一致性的优化，克服多尺度特征预测中因上采样导致的视差错位问题。
• 开发一种单阶段、高效的蒸馏框架，避免先前蒸馏方法存在的两阶段训练开销。

提出的方法

• 提出自参考蒸馏，即教师模型与学生模型采用相同架构，并在每个训练周期通过学生模型权重进行更新，以提供持续演化的监督信号。
• 应用多视角一致性检查过滤器，通过验证前向与后向图像扭曲之间的一致性，检测并抑制不可靠的深度预测结果——尤其在运动或边缘区域。
• 利用高层与低层特征之间的上下文一致性，预测多尺度视差偏移量，以逐步优化视差图。
• 利用视差偏移场纠正上采样过程中引入的错位问题，提升不同特征尺度间的对齐精度。
• 端到端训练学生模型，损失函数结合光度重建损失与来自过滤后教师输出的蒸馏损失。
• 采用由多视角一致性检查生成的二值化硬掩码，选择性抑制高误差区域的蒸馏过程，提升模型鲁棒性。

Figure 1: Illustration of the depth map, error map and binary mask from the proposed method. The first row is the RGB image, and the second row is the depth map. The third row is the intermediate error map generated by the multiview check filter, where the places with large errors are red and yellow

实验结果

研究问题

• RQ1 持续更新的教师模型是否能在无需单独预训练的情况下，提升自监督深度估计性能？
• RQ2 多视角一致性检查在蒸馏过程中过滤教师模型不可靠深度预测的效率如何？
• RQ3 基于特征层级上下文的视差偏移优化是否能有效减少多尺度视差预测中的错位问题？
• RQ4 所提方法是否在性能与效率上优于现有两阶段蒸馏方法？

主要发现

• 所提方法在KITTI与Make3D数据集上达到最先进性能，优于以往自监督方法。
• 自参考蒸馏模块显著提升了Sq Rel指标，证明其在降低深度估计误差方面的有效性。
• 多视角检查过滤器生成的硬掩码在所有评估指标中表现优于软掩码、发现掩码与自动掩码，尤其在处理异常值方面优势明显。
• 采用MPViT-S作为骨干网络时性能最佳，表明所提模块具有强大的特征表征学习能力。
• 该方法以极低计算开销实现优越性能，在训练时间与FLOPs方面均优于两阶段蒸馏方法。
• 消融实验证实，自参考蒸馏、多视角检查与视差对齐三个模块均独立且显著贡献于性能提升。

Figure 2: Overview of the proposed framework. The model adopts an encoder-decoder architecture, in which the encoder is a commonly used backbone, such as ResNet [ 13 ] and Swin Transformer [ 14 ] . Our decoder outputs disparity, which is then converted to depth. The branch of the teacher model provi