[论文解读] ACORN: Adaptive Coordinate Networks for Neural Scene Representation

Julien Martel, David B. Lindell+4 more|arXiv (Cornell University)|May 6, 2021

3D Shape Modeling and Analysis参考文献 4被引用 12

一句话总结

ACORN 提出了一种混合隐式-显式神经网络架构，通过自适应地将二维和三维场景划分为多尺度块，利用坐标编码器生成特征网格，并通过轻量级解码器实现高效推理。该方法首次实现了约 40 dB PSNR 的千兆像素图像拟合，并将 3D 形状训练时间从数天缩短至数分钟，内存使用减少 10 倍以上。

ABSTRACT

Neural representations have emerged as a new paradigm for applications in rendering, imaging, geometric modeling, and simulation. Compared to traditional representations such as meshes, point clouds, or volumes they can be flexibly incorporated into differentiable learning-based pipelines. While recent improvements to neural representations now make it possible to represent signals with fine details at moderate resolutions (e.g., for images and 3D shapes), adequately representing large-scale or complex scenes has proven a challenge. Current neural representations fail to accurately represent images at resolutions greater than a megapixel or 3D scenes with more than a few hundred thousand polygons. Here, we introduce a new hybrid implicit-explicit network architecture and training strategy that adaptively allocates resources during training and inference based on the local complexity of a signal of interest. Our approach uses a multiscale block-coordinate decomposition, similar to a quadtree or octree, that is optimized during training. The network architecture operates in two stages: using the bulk of the network parameters, a coordinate encoder generates a feature grid in a single forward pass. Then, hundreds or thousands of samples within each block can be efficiently evaluated using a lightweight feature decoder. With this hybrid implicit-explicit network architecture, we demonstrate the first experiments that fit gigapixel images to nearly 40 dB peak signal-to-noise ratio. Notably this represents an increase in scale of over 1000x compared to the resolution of previously demonstrated image-fitting experiments. Moreover, our approach is able to represent 3D shapes significantly faster and better than previous techniques; it reduces training times from days to hours or minutes and memory requirements by over an order of magnitude.

研究动机与目标

• 为解决现有神经场景表征在处理大规模或复杂 3D 场景及高分辨率图像时的可扩展性限制。
• 通过引入混合隐式-显式架构，克服神经表征中计算效率与内存使用之间的权衡。
• 在训练和推理过程中基于局部信号复杂度实现自适应资源分配，从而提升准确性和效率。
• 通过神经网络实现高分辨率图像和复杂 3D 几何体拟合的最先进性能。

提出的方法

• 该方法采用类似于四叉树或八叉树的多尺度块坐标分解，在训练过程中优化以自适应地细分信号域。
• 坐标编码器通过单次前向传播为每个块生成低分辨率特征网格，占网络参数的大部分。
• 轻量级特征解码器通过插值实现对每个块内连续坐标的快速高效评估。
• 网络通过自定义训练流程进行端到端优化，联合学习块划分与网络权重，采用整数线性规划求解。
• 该架构通过将特征网格与连续解码相结合，同时支持隐式（基于坐标）和显式（基于网格）表征。
• 在训练过程中动态调整资源分配，将更多块分配给局部复杂度较高的区域。

实验结果

研究问题

• RQ1 神经场景表征能否在保持高保真度和高效推理的前提下扩展至千兆像素图像？
• RQ2 在复杂 3D 场景建模中，神经表征如何在内存效率与计算速度之间实现平衡？
• RQ3 与固定网格或均匀分辨率方法相比，自适应块划分是否能提升表征精度和训练效率？
• RQ4 混合隐式-显式架构在训练时间、内存使用和重建质量方面，相较于纯隐式或显式神经表征，能实现多大程度的性能超越？

主要发现

• ACORN 首次成功将千兆像素图像（8192×8192）拟合至神经表征，峰值信噪比（PSNR）约为 40 dB。
• 该方法将 3D 形状表征的训练时间从数天缩短至数分钟，相比先前方法内存使用减少 10 倍以上。
• 对于如 Lucy（1400 万顶点）等 3D 模型，ACORN 实现的模型大小仅为 68 MB，显著小于未压缩的 1.2 GB 和压缩后的 380 MB 版本。
• 尽管未显式强制连续性，该方法在块边界处仍表现出高保真度拟合，仅在少数情况下观察到轻微伪影。
• 与先前仅限于子兆像素分辨率的图像拟合实验相比，ACORN 的分辨率规模提升了 1000 倍。