[论文解读] SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates

Zhizhong Han, Guanhui Qiao+2 more|arXiv (Cornell University)|Mar 12, 2020

3D Shape Modeling and Analysis参考文献 61被引用 4

一句话总结

该论文提出SeqXY2SeqZ，一种新颖的3D形状结构学习方法，通过将形状表示为从2D坐标映射到第三维上1D占据段序列的2D函数，避免了密集的3D采样。采用基于RNN的Seq2Seq模型并结合注意力机制，实现了SOTA级别的重建性能，复杂度为O(R²)，相较于3D隐式函数方法显著降低了内存占用和推理时间。

ABSTRACT

Structure learning for 3D shapes is vital for 3D computer vision. State-of-the-art methods show promising results by representing shapes using implicit functions in 3D that are learned using discriminative neural networks. However, learning implicit functions requires dense and irregular sampling in 3D space, which also makes the sampling methods affect the accuracy of shape reconstruction during test. To avoid dense and irregular sampling in 3D, we propose to represent shapes using 2D functions, where the output of the function at each 2D location is a sequence of line segments inside the shape. Our approach leverages the power of functional representations, but without the disadvantage of 3D sampling. Specifically, we use a voxel tubelization to represent a voxel grid as a set of tubes along any one of the X, Y, or Z axes. Each tube can be indexed by its 2D coordinates on the plane spanned by the other two axes. We further simplify each tube into a sequence of occupancy segments. Each occupancy segment consists of successive voxels occupied by the shape, which leads to a simple representation of its 1D start and end location. Given the 2D coordinates of the tube and a shape feature as condition, this representation enables us to learn 3D shape structures by sequentially predicting the start and end locations of each occupancy segment in the tube. We implement this approach using a Seq2Seq model with attention, called SeqXY2SeqZ, which learns the mapping from a sequence of 2D coordinates along two arbitrary axes to a sequence of 1D locations along the third axis. SeqXY2SeqZ not only benefits from the regularity of voxel grids in training and testing, but also achieves high memory efficiency. Our experiments show that SeqXY2SeqZ outperforms the state-ofthe-art methods under widely used benchmarks.

研究动机与目标

• 解决基于隐式函数的3D形状重建中密集3D采样的高计算与内存开销问题。
• 通过利用结构化的2D函数表示，消除训练和推理过程中的不规则3D采样。
• 利用内存高效的基于序列的深度学习模型，实现高效、高分辨率的3D形状重建。
• 在3D形状重建基准测试中实现SOTA性能，同时提升推理速度并降低内存使用。

提出的方法

• 将3D体素网格表示为沿X、Y或Z轴的1D管道集合，由正交平面上的2D坐标索引。
• 将每条管道简化为一系列占据段，每个占据段由起始和结束的1D体素位置定义。
• 使用带有双向RNN编码器和自回归解码器的Seq2Seq模型，从2D坐标和形状条件预测1D占据段序列。
• 采用注意力机制，使解码器在序列预测段边界时能够聚焦于相关2D坐标。
• 利用体素管道化技术，将立方体3D复杂度转化为二维的O(R²)复杂度，实现高效学习与推理。
• 通过在3D体素网格上使用自编码损失进行端到端训练，将形状特征作为管道预测的上下文。

实验结果

研究问题

• RQ1 能否通过从2D坐标预测1D占据段来有效学习3D形状结构，从而避免密集的3D采样？
• RQ2 将体素网格表示为映射到1D段序列的2D函数，是否能提升内存效率和推理速度？
• RQ3 带有注意力机制的Seq2Seq模型能否有效学习并从2D坐标输入重建复杂3D形状？
• RQ4 与基于3D隐式函数的模型相比，该方法在重建质量、内存使用和推理时间方面表现如何？

主要发现

• SeqXY2SeqZ在广泛使用的基准数据集上实现了SOTA级别的3D形状重建性能，优于现有方法。
• 对于分辨率R³的网格，模型仅需O(R²)次RNN推理步骤，相比3D隐式函数的O(R³)采样，计算复杂度显著降低。
• 在R=64时，重建仅需286 MB内存，远优于DISN（>11 GB）和OccNet（1175 MB）。
• CPU上的推理时间缩短至8.79秒，优于OccNet（55.80秒）和DISN（14.68秒）在相同分辨率下的表现。
• 该方法表现出高效率，每2D坐标仅需1至3个占据段即可重建如椅子和桌子等复杂形状。
• 注意力可视化结果表明，模型学习到了有意义的空间依赖关系，简单形状（如汽车）的注意力模式更简单。