[论文解读] Semantic Scene Completion via Integrating Instances and Scene in-the-Loop

Yingjie Cai, Xuesong Chen+4 more|arXiv (Cornell University)|Apr 8, 2021

Advanced Vision and Imaging参考文献 43被引用 7

一句话总结

该论文提出SISNet，一种新颖的迭代框架，通过在场景到实例与实例到场景的语义补全之间交替进行，以提升3D语义场景补全性能。通过将实例从粗粒度完成的场景中解耦，并在参数共享的迭代循环中同时利用场景和实例级别的先验知识，SISNet在NYU、NYUCAD和SUNCG-RGBD数据集上实现了最先进性能，显著提升了细粒度形状恢复效果并减少了类别混淆。

ABSTRACT

Semantic Scene Completion aims at reconstructing a complete 3D scene with precise voxel-wise semantics from a single-view depth or RGBD image. It is a crucial but challenging problem for indoor scene understanding. In this work, we present a novel framework named Scene-Instance-Scene Network ( extit{SISNet}), which takes advantages of both instance and scene level semantic information. Our method is capable of inferring fine-grained shape details as well as nearby objects whose semantic categories are easily mixed-up. The key insight is that we decouple the instances from a coarsely completed semantic scene instead of a raw input image to guide the reconstruction of instances and the overall scene. SISNet conducts iterative scene-to-instance (SI) and instance-to-scene (IS) semantic completion. Specifically, the SI is able to encode objects' surrounding context for effectively decoupling instances from the scene and each instance could be voxelized into higher resolution to capture finer details. With IS, fine-grained instance information can be integrated back into the 3D scene and thus leads to more accurate semantic scene completion. Utilizing such an iterative mechanism, the scene and instance completion benefits each other to achieve higher completion accuracy. Extensively experiments show that our proposed method consistently outperforms state-of-the-art methods on both real NYU, NYUCAD and synthetic SUNCG-RGBD datasets. The code and the supplementary material will be available at \url{https://github.com/yjcaimeow/SISNet}.

研究动机与目标

• 为解决从单视角RGBD输入中获取的3D场景理解不完整且模糊的问题，特别是针对细粒度形状细节和语义混淆区域。
• 克服现有SSC和SIC方法的局限性，这些方法或忽略实例级别约束，或未能有效利用场景上下文以提升实例检测与补全效果。
• 开发一种统一框架，实现场景与实例层级之间的双向信息流动，以提升整体补全精度。
• 通过迭代、参数高效的机制整合场景与实例级别的语义先验，实现在基准数据集上的最先进性能。

提出的方法

• 该框架首先利用多模态输入（2D语义图与TSDF）生成一个粗粒度完成的3D语义场景，形成初始场景$S_0$。
• 场景到实例（SI）补全在$S_0$中定位物体，并以更高分辨率将每个实例体素化，以恢复细粒度3D形状。
• 实例到场景（IS）补全将优化后的实例形状重新整合回场景中，通过上下文反馈提升整体场景补全质量。
• 该方法采用迭代、参数共享机制，在SI与IS补全之间交替进行，实现场景与实例预测之间的相互增强。
• 迭代过程通过多阶段监督端到端训练，使模型能够逐步优化实例细节与全局场景语义。
• 该框架利用类别先验与场景上下文，解决物体类别中的歧义（如窗户与墙壁的混淆），并提升检测精度。

Figure 2: Overview of the Proposed Method . SISNet consists of iterative scene-to-instance completion and instance-to-scene completion stages. Given single-view RGBD images, TSDF from the depth map and semantic volume from the reprojection of 2D semantic segmentation are input into the initial scene

实验结果

研究问题

• RQ1 将实例级别细节整合到粗粒度完成的场景中，是否能提升3D语义场景补全的准确性？
• RQ2 在场景与实例层级之间进行迭代优化，是否能优于单阶段或非迭代方法？
• RQ3 当作为监督信号使用时，场景上下文是否能显著提升实例检测与形状补全效果？
• RQ4 在处理复杂、遮挡或语义模糊区域时，该方法与最先进SSC和SIC方法相比表现如何？
• RQ5 在场景-实例-场景循环中，最优迭代次数是多少，能够在性能与计算成本之间实现最佳平衡？

主要发现

• 在NYU数据集上，SISNet在一次迭代后相较基线$S_0$的SSC IoU提升5.2%，两次迭代后进一步提升2%。
• 在NYUCAD数据集上，该方法一次迭代提升SSC IoU 4.4%，两次迭代再提升2%，表明在各数据集上均具有一致增益。
• 在SUNCG-RGBD数据集上，SISNet在SC IoU上较Sketch高出8.1%，在SSC IoU上高出23.2%，且计算成本相近。
• 消融实验表明，跳过实例补全部分会平均使SSC性能下降2.8%，证明其在优化过程中的关键作用。
• 初始场景补全步骤使mAP与召回率提升约10%，表明场景上下文能有效增强实例定位能力。
• 超过两次迭代后性能无显著提升，表明两次迭代已足够实现最优性能。

Figure 3: Architecture details of initial scene completion. Given TSDF and semantic volume, to infer missing invisible shape details, we exploit DDR to encode features and propagate information with skip layers, yielding a whole scene completion results.