[论文解读] PlenOctrees for Real-time Rendering of Neural Radiance Fields

Alex Yu, Ruilong Li+4 more|arXiv (Cornell University)|Mar 25, 2021

Computer Graphics and Visualization Techniques被引用 13

一句话总结

本文提出 PlenOctrees，一种基于层次八叉树的3D表示方法，通过将NeRF预测预先查表至视图相关的体素结构，实现了超过150 FPS的神经辐射场（NeRF）实时渲染。通过训练一种改进的NeRF以预测视图相关的外观球谐系数，并优化PlenOctree结构，该方法在不损失图像质量的前提下实现了实时性能，支持交互式6-DOF查看和浏览器内渲染。

ABSTRACT

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu.net/plenoctrees

研究动机与目标

• 为解决NeRF渲染速度慢这一关键瓶颈，该瓶颈限制了实时交互应用的使用。
• 在实时渲染流水线中保留镜面高光等视图相关效果。
• 通过将NeRF模型早期转换为PlenOctree表示，加速NeRF训练。
• 实现在消费级硬件上对照片级真实感场景的交互式6-DOF渲染，包括浏览器内部署。
• 在与原始NeRF相比质量损失极小的前提下实现高保真渲染。

提出的方法

• 该方法引入PlenOctrees，一种稀疏的、分层的八叉树结构，其中每个叶节点存储用于视图相关辐射度的密度和球谐（SH）系数。
• 训练一种改进的NeRF网络以预测SH系数而非原始RGB值，从而实现高效地查表至PlenOctree。
• 在渲染过程中，通过在任意视图方向上评估SH函数来建模视图相关的外观。
• 通过可微分渲染对PlenOctree结构进行端到端优化，使重建质量优于原始NeRF。
• 通过提前停止NeRF训练并转为PlenOctree优化，加速训练过程，避免昂贵的神经网络推理。
• 通过实现兼容WebGL的片段着色器，实现在消费级笔记本电脑上实时渲染PlenOctrees的浏览器内渲染。

Figure 1: Real-time NeRF with PlenOctrees. Given a set of posed images of a scene, our method creates a 3D volumetric model that can be rendered in real-time. We propose PlenOctrees, which are octrees that can capture view-dependent dependent effects such as specularities. Rendering using our approa

实验结果

研究问题

• RQ1 能否在保留镜面高光等视图相关效果的同时实现实时渲染NeRF？
• RQ2 像PlenOctree这样的分层体素表示能否在渲染速度和图像质量方面均优于标准NeRF？
• RQ3 将NeRF预测预先查表至PlenOctree是否能实现比完整NeRF训练更快的训练速度和收敛速度？
• RQ4 PlenOctrees能否有效用于交互式6-DOF场景探索和浏览器内渲染？
• RQ5 使用球谐函数作为外观建模的基函数，能否在PlenOctree中实现准确、高效且可微分的视图相关渲染？

主要发现

• 在V100 GPU上，该方法实现800×800图像渲染的帧率为167.68 FPS，相比传统NeRF提速达3000倍。
• PlenOctrees生成的图像质量与原始NeRF相当或更优，经基准数据集上的定量与定性比较验证。
• 将NeRF-SH模型早期转换为PlenOctree并进行后续优化，可显著减少有效训练时间：4.5小时的PlenOctree优化可达到16小时完整NeRF训练的质量。
• 浏览器内WebGL渲染器使消费级笔记本电脑能够实现实时6-DOF交互，展示了实际部署潜力。
• 平均未压缩PlenOctree大小为1.93 GB（合成数据）和3.53 GB（Tanks and Temples），但压缩版本小至30–120 MB，适合在线分发。
• 该方法在真实场景和前向视角场景中表现良好，尽管未针对无限场景或高度前向视角场景进行优化，此类场景下MPI可能更合适。

Figure 2: Method Overview. We propose a method to quickly render NeRFs by training a modified NeRF model (NeRF-SH) and converting it into a PlenOctree, an octree that captures view-dependent effects. a) The NeRF-SH model uses the same optimization procedure and volume rendering method presented in N