[论文解读] Self-Supervised Learning of Depth and Camera Motion from 360° Videos

Fu-En Wang, Hou-Ning Hu+6 more|arXiv (Cornell University)|Nov 13, 2018

Advanced Vision and Imaging参考文献 28被引用 11

一句话总结

该论文提出了一种用于360°视频的自监督单目深度与相机运动估计方法，通过立方体贴图投影消除等距圆柱投影失真。该方法引入了球面光度一致性损失和位姿一致性损失以提高精度，在新的PanoSUNCG数据集上实现了最先进性能，且推理速度优于等距圆柱基线方法。

ABSTRACT

As 360° cameras become prevalent in many autonomous systems (e.g., self-driving cars and drones), efficient 360° perception becomes more and more important. We propose a novel self-supervised learning approach for predicting the omnidirectional depth and camera motion from a 360° video. In particular, starting from the SfMLearner, which is designed for cameras with normal field-of-view, we introduce three key features to process 360° images efficiently. Firstly, we convert each image from equirectangular projection to cubic projection in order to avoid image distortion. In each network layer, we use Cube Padding (CP), which pads intermediate features from adjacent faces, to avoid image boundaries. Secondly, we propose a novel "spherical" photometric consistency constraint on the whole viewing sphere. In this way, no pixel will be projected outside the image boundary which typically happens in images with normal field-of-view. Finally, rather than naively estimating six independent camera motions (i.e., naively applying SfM-Learner to each face on a cube), we propose a novel camera pose consistency loss to ensure the estimated camera motions reaching consensus. To train and evaluate our approach, we collect a new PanoSUNCG dataset containing a large amount of 360° videos with groundtruth depth and camera motion. Our approach achieves state-of-the-art depth prediction and camera motion estimation on PanoSUNCG with faster inference speed comparing to equirectangular. In real-world indoor videos, our approach can also achieve qualitatively reasonable depth prediction by acquiring model pre-trained on PanoSUNCG.

研究动机与目标

• 解决自动驾驶系统中360°视频缺乏有效深度与运动估计方法的问题。
• 克服全景视频处理中固有的等距圆柱投影几何失真问题。
• 开发一种仅依赖视频序列而无需显式监督的自监督学习框架。
• 通过利用360°视频的球面几何结构提高深度与运动估计的准确性。
• 实现实时高效推理，适用于机器人和自动驾驶系统的部署。

提出的方法

• 将360°等距圆柱图像转换为立方体贴图投影，以减少失真并提高特征学习效率。
• 提出立方体填充（CP）方法，以处理网络推理过程中相邻立方体面之间的特征边界问题。
• 提出一种球面光度一致性损失，计算整个视球面上的重建误差，避免边界投影问题。
• 设计一种新颖的相机位姿一致性损失，强制六张立方体面之间的一致性，提升运动估计的鲁棒性。
• 使用视频帧的自监督方式端到端训练模型，最小化光度重建误差。
• 在真实世界360°视频上对预训练模型进行微调，实现零样本泛化。

实验结果

研究问题

• RQ1 能否将自监督深度与运动估计有效扩展至具有球面几何结构的360°视频？
• RQ2 与等距圆柱投影相比，基于立方体贴图的处理在准确性和推理速度方面表现如何？
• RQ3 球面光度一致性损失是否能在360°视频任务中超越传统平面损失？
• RQ4 在立方体面之间强制位姿一致性是否能提升运动估计的鲁棒性？
• RQ5 在合成数据上预训练的模型是否能泛化到真实世界360°视频场景？

主要发现

• 所提方法在PanoSUNCG数据集上实现了最先进深度预测性能，绝对相对误差（Abs Rel）为0.337。
• 该方法将RMSE降低至8.589，并将δ<1.25指标提升至0.647，表明在所有阈值下均具有更优的深度准确性。
• 相机运动估计的旋转误差为6.98°，平移误差为0.025，优于基线方法及消融实验变体。
• 同时包含球面光度一致性损失与位姿一致性损失的完整模型显著优于缺少这些组件的消融实验。
• 由于立方体贴图表示减少了计算负载，该方法在高分辨率下的推理速度比等距圆柱基线快超过200%。
• 模型在微调RICOH THETA V拍摄的真实360°视频后，能很好地泛化到真实世界场景，生成定性合理的深度预测结果。