[论文解读] Scalable Semantic 3D Mapping of Coral Reefs with Deep Learning

Jonathan Sauder, Guilhem Banc‐Prandi+2 more|arXiv (Cornell University)|Sep 22, 2023

3D Surveying and Cultural Heritage被引用 5

一句话总结

本文提出了一种可扩展的、低成本的流水线，用于使用自运动视频实现珊瑚礁的自动化语义3D制图，结合基于学习的运动结构（SfM）与深度语义分割。该方法在5分钟内完成100米视频剖面的处理，生成高精度的3D点云并附带密集的语义标签，显著降低了人工与计算成本，同时实现了大规模珊瑚礁监测。

ABSTRACT

Coral reefs are among the most diverse ecosystems on our planet, and are depended on by hundreds of millions of people. Unfortunately, most coral reefs are existentially threatened by global climate change and local anthropogenic pressures. To better understand the dynamics underlying deterioration of reefs, monitoring at high spatial and temporal resolution is key. However, conventional monitoring methods for quantifying coral cover and species abundance are limited in scale due to the extensive manual labor required. Although computer vision tools have been employed to aid in this process, in particular SfM photogrammetry for 3D mapping and deep neural networks for image segmentation, analysis of the data products creates a bottleneck, effectively limiting their scalability. This paper presents a new paradigm for mapping underwater environments from ego-motion video, unifying 3D mapping systems that use machine learning to adapt to challenging conditions under water, combined with a modern approach for semantic segmentation of images. The method is exemplified on coral reefs in the northern Gulf of Aqaba, Red Sea, demonstrating high-precision 3D semantic mapping at unprecedented scale with significantly reduced required labor costs: a 100 m video transect acquired within 5 minutes of diving with a cheap consumer-grade camera can be fully automatically analyzed within 5 minutes. Our approach significantly scales up coral reef monitoring by taking a leap towards fully automatic analysis of video transects. The method democratizes coral reef transects by reducing the labor, equipment, logistics, and computing cost. This can help to inform conservation policies more efficiently. The underlying computational method of learning-based Structure-from-Motion has broad implications for fast low-cost mapping of underwater environments other than coral reefs.

研究动机与目标

• 解决因人工分析照片样方和视频剖面而造成的珊瑚礁监测可扩展性瓶颈。
• 通过将基于学习的SfM与深度学习相结合，克服传统SfM和语义分割的局限，实现端到端的3D语义制图。
• 通过低代码、全自动的流水线，减少对高端设备、专家人力和复杂后勤的依赖，实现珊瑚礁监测的普及化。
• 支持在气候脆弱区域（如阿卡巴湾北部）实现高分辨率、大规模的珊瑚礁生态监测，以支持保护政策与韧性评估。
• 为利用可及的视频数据与深度学习技术，建立可扩展的、自动化水下环境3D制图基础，超越珊瑚礁应用。

提出的方法

• 使用佩戴在潜水员身上的消费级运动相机采集自运动视频，捕捉珊瑚礁场景，避免对GPS或专用硬件的依赖。
• 采用基于学习的运动结构（SfM）系统，从单目视频中估计3D几何结构与相机位姿，实现实时重建，无需回环检测或全局优化。
• 集成深度神经网络进行单目深度估计与视觉里程计，从视频序列中生成精确的3D点云。
• 应用最先进的语义分割模型对密集视频帧进行处理，高精度分类底栖特征（如珊瑚、沙地与藻类）。
• 将预测的3D几何结构与语义标签相结合，生成语义3D点云，其中每个点均标注其底栖类别。
• 采用超分辨率技术优化深度估计，提升几何精度，补偿深度学习推理中的分辨率限制。

Figure 1 : Existing conventional SfM fails to produce a coherent point cloud from uncurated image collections such as video frames. This example shows the point clouds from a video transect in the King Abdullah Reef in Aqaba, Jordan. Leftmost panel: our proposed method creates a coherent point cloud

实验结果

研究问题

• RQ1 仅使用自运动视频，完全自动化的基于学习的SfM流水线能否达到生态监测珊瑚礁所需的几何精度？
• RQ2 在光照与水下条件多变的情况下，深度语义分割模型在水下视频帧中对底栖类别的分类准确度如何？
• RQ3 与传统的人工或半自动方法相比，该流水线在将原始视频转换为语义3D点云方面的可扩展性与效率如何？
• RQ4 该方法能否在无GPS或回环检测的情况下，仅依赖可见的剖面标记实现地理上准确的3D重建？
• RQ5 该框架在扩展至其他水下生态系统（如深海探测）方面具有多大潜力？

主要发现

• 仅用5分钟潜水采集的100米视频剖面，可在5分钟内通过该流水线完全处理为语义3D点云。
• 该方法实现了高空间精度与优异的语义分割性能，支持对底栖覆盖度与物种分布的精确量化。
• 与传统照片样方或人工视频分析方法相比，该方法显著降低了人工成本与后勤复杂度。
• 作者发布了来自阿卡巴湾北部的大量自运动视频数据集，以及用于底栖分割的密集标注视频帧基准数据集。
• 该方法在低能见度与光照多变等挑战性水下条件下表现出强鲁棒性，无需高端传感器或GPS支持。
• 只要具备足够的标注视频数据，该框架可推广至其他水下环境（如红树林森林或深海区域）。

Figure 2 : Example excerpts of 3D point clouds of different reef scenarios in their original RGB color, next to the points colorized by their predicted benthic class (top). A 100 m transect (bottom) can be covered by a diver in less than five minutes: the length of the created point clouds is limite