[Paper Review] TriVol: Point Cloud Rendering via Triple Volumes

Tao Hu, Xiaogang Xu+2 more|arXiv (Cornell University)|Mar 29, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes TriVol, a lightweight yet dense 3D representation composed of three slim, axis-aligned volumes derived from point clouds, enabling high-resolution, view-consistent, and photo-realistic rendering via NeRF. By fusing multi-scale features through efficient 3D ConvNets and trilinear interpolation, TriVol achieves state-of-the-art rendering quality with reduced computational cost and strong zero-shot generalization to unseen objects of the same category.

ABSTRACT

Existing learning-based methods for point cloud rendering adopt various 3D representations and feature querying mechanisms to alleviate the sparsity problem of point clouds. However, artifacts still appear in rendered images, due to the challenges in extracting continuous and discriminative 3D features from point clouds. In this paper, we present a dense while lightweight 3D representation, named TriVol, that can be combined with NeRF to render photo-realistic images from point clouds. Our TriVol consists of triple slim volumes, each of which is encoded from the point cloud. TriVol has two advantages. First, it fuses respective fields at different scales and thus extracts local and non-local features for discriminative representation. Second, since the volume size is greatly reduced, our 3D decoder can be efficiently inferred, allowing us to increase the resolution of the 3D space to render more point details. Extensive experiments on different benchmarks with varying kinds of scenes/objects demonstrate our framework's effectiveness compared with current approaches. Moreover, our framework has excellent generalization ability to render a category of scenes/objects without fine-tuning.

Motivation & Objective

• To address view inconsistency and hole artifacts in existing point cloud rendering methods.
• To overcome the inefficiency of dense 3D feature volumes in 3D ConvNets for high-resolution rendering.
• To design a lightweight, discriminative 3D representation that supports continuous feature querying across all 3D locations.
• To enable zero-shot generalization to unseen scenes or objects of the same category without fine-tuning.
• To combine the TriVol representation with NeRF for photo-realistic rendering with high detail and minimal artifacts.

Proposed method

• TriVol is constructed by voxelizing the input point cloud and reorganizing voxels along each of the three spatial axes into three separate, slim 3D volumes.
• An efficient encoder uses a simple grouping strategy—without additional neural networks—to generate the Initial TriVol from the point cloud.
• A decoder applies three independent 3D UNets to transform each slim volume into a dense feature representation, enabling high-resolution feature extraction.
• The Feature TriVol allows trilinear interpolation-based feature querying at any 3D point, ensuring continuous and discriminative features.
• The framework integrates TriVol with the NeRF rendering pipeline to generate photo-realistic images with sharp details and no holes.
• The method supports high-resolution 3D space (e.g., 256³) with significantly reduced memory and FLOPS compared to dense voxel grids.

Figure 2 : Overview of the proposed TriVol for point cloud rendering. (a) Encoding : Input point cloud is encoded to our Initial TriVol along $x$ , $y$ , and $z$ axis; (b) Decoding : Each volume is decoded to dense feature volume via a unique 3D UNet; (c) Feature Querying : Any point’s feature is qu

Experimental results

Research questions

• RQ1 Can a lightweight 3D representation be designed to support high-resolution, continuous feature extraction from sparse point clouds?
• RQ2 How can multi-scale local and non-local features be effectively fused in a 3D representation for improved rendering quality?
• RQ3 Can the proposed method achieve view-consistent and photo-realistic rendering without requiring fine-tuning on new scenes or objects?
• RQ4 How does the TriVol representation compare in efficiency and accuracy to dense voxel-based and 2D-based rendering methods?
• RQ5 To what extent does the method generalize to point clouds with varying point densities or unseen categories?

Key findings

• TriVol achieves a PSNR of 27.22 on the ShapeNet dataset with 100k training and testing points, outperforming Point-NeRF (25.73) and other SOTA methods.
• With a resolution of 256³ and group size G=16, TriVol achieves 27.22 PSNR while using only 8.47 GB GPU memory, avoiding OOM issues seen with 256³ dense voxels.
• The method maintains strong performance under distribution shift: when testing with 10k points after training on 100k points, it achieves 27.01 PSNR, outperforming Point-NeRF (22.75).
• TriVol demonstrates superior generalization: it renders high-quality images for unseen objects of the same category without fine-tuning, due to its discriminative and continuous 3D features.
• The framework reduces FLOPS by up to 95% compared to dense 128³ voxels while improving PSNR by over 1.69 points.
• The ablation study confirms that the grouping-based encoder is more effective than standard point-based backbones, achieving 18.56 PSNR on ScanNet, a 0.15 improvement over baseline.

Figure 3 : Qualitative point cloud rendering comparison between ours and SOTA methods and baselines on the ScanNet dataset.