[论文解读] Evaluating Point Cloud from Moving Camera Videos: A No-Reference Metric

Zicheng Zhang, Wei Sun+5 more|arXiv (Cornell University)|Aug 30, 2022

3D Shape Modeling and Analysis被引用 4

一句话总结

该论文提出了一种无参考点云质量评估方法，通过分析点云的移动摄像机视频来评估视觉质量。通过使用ResNet50和SlowFast R50模型提取空间（静态）和时间（动态）质量感知特征，该方法实现了最先进性能，在SJTU-PCQA数据集上的SRCC高达0.8611，与全参考方法相当，并优于现有无参考方法。

ABSTRACT

Point cloud is one of the most widely used digital representation formats for three-dimensional (3D) contents, the visual quality of which may suffer from noise and geometric shift distortions during the production procedure as well as compression and downsampling distortions during the transmission process. To tackle the challenge of point cloud quality assessment (PCQA), many PCQA methods have been proposed to evaluate the visual quality levels of point clouds by assessing the rendered static 2D projections. Although such projection-based PCQA methods achieve competitive performance with the assistance of mature image quality assessment (IQA) methods, they neglect that the 3D model is also perceived in a dynamic viewing manner, where the viewpoint is continually changed according to the feedback of the rendering device. Therefore, in this paper, we evaluate the point clouds from moving camera videos and explore the way of dealing with PCQA tasks via using video quality assessment (VQA) methods. First, we generate the captured videos by rotating the camera around the point clouds through several circular pathways. Then we extract both spatial and temporal quality-aware features from the selected key frames and the video clips through using trainable 2D-CNN and pre-trained 3D-CNN models respectively. Finally, the visual quality of point clouds is represented by the video quality values. The experimental results reveal that the proposed method is effective for predicting the visual quality levels of the point clouds and even competitive with full-reference (FR) PCQA methods. The ablation studies further verify the rationality of the proposed framework and confirm the contributions made by the quality-aware features extracted via the dynamic viewing manner. The code is available at https://github.com/zzc-1998/VQA_PC.

研究动机与目标

• 为解决现有基于投影的无参考PCQA方法仅依赖静态二维视图的局限性。
• 探究动态观看行为（模拟移动摄像机）是否能提升点云质量评估效果。
• 开发一种利用视频质量评估（VQA）建模点云中空间与时间失真的无参考PCQA框架。
• 验证使用多条摄像机路径捕获多样化动态视图以提升质量预测性能的有效性。
• 在保持高精度的同时确保计算效率，以支持实际部署。

提出的方法

• 通过在点云周围沿四条不同圆形路径旋转虚拟摄像机，生成视频序列。
• 使用可训练的2D-CNN（ResNet50）从关键帧中提取空间质量特征，以捕捉噪声和几何偏移等静态失真。
• 使用预训练的3D-CNN（SlowFast R50）从视频片段中提取时间质量特征，以建模如视图突变等运动相关伪影。
• 最终质量分数通过空间与时间特征的融合获得，使模型能够同时评估静态与动态视觉质量。
• 该框架通过回归头端到端训练，以预测主观质量分数，无需原始无损参考点云。
• 在三个基准数据集（SJTU-PCQA、WPC和LSPCQA-I）上进行评估，主要使用SRCC和PLCC作为指标。

Figure 1: Illustration of human habits of observing point clouds via static and dynamic views. The static views focus on the single rendered images while the dynamic views focus on the frames (usually generated by rotation).

实验结果

研究问题

• RQ1 与基于静态投影的方法相比，通过移动摄像机视频建模动态观看行为是否能提升无参考点云质量评估性能？
• RQ2 从视频序列中提取的空间与时间特征如何共同贡献于整体质量预测性能？
• RQ3 不同摄像机路径在捕获与质量相关的视觉内容方面有何相对贡献？
• RQ4 与现有无参考及全参考PCQA方法相比，所提方法在性能与效率方面表现如何？
• RQ5 所提出的基于VQA的框架是否能在不访问原始点云的情况下，实现与全参考方法相当的性能？

主要发现

• 在SJTU-PCQA数据集上，该方法实现了0.8611的SRCC和0.8702的PLCC，优于所有现有无参考PCQA方法。
• 该方法在性能上与全参考PCQA方法相当，表明基于视频的动态评估比静态投影更能有效捕捉感知质量。
• 消融实验证实，空间与时间特征均至关重要，其中时间分支显著提升了仅依赖静态特征基线的性能。
• 使用全部四条摄像机路径（P7）的模型达到最高性能，且移除任意一条路径均导致可测量的性能下降，证实了多路径视频捕获的价值。
• 尽管包含视频生成时间，该方法仍保持了良好的计算效率，平均每帧点云的推理时间为13.15秒，快于大多数基于模型的PCQA方法。
• 路径θ_A对性能贡献最大，如P6模型（排除θ_A）性能最低所示，表明不同路径对特定失真类型具有敏感性。

Figure 3: The framework of the proposed method, which includes the video capture process, the feature extraction module, and the feature regression module.