[论文解读] LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering

Ricardo Martin-Brualla, Rohit Pandey+15 more|arXiv (Cornell University)|Nov 12, 2018

Advanced Vision and Imaging参考文献 6被引用 16

一句话总结

LookinGood 引入了一种实时神经重渲染系统，通过使用深度学习模型联合完成补全、超分辨率和去噪，从而提升动作捕捉管道生成的低质量输出。该方法采用以语义显著性为中心并具备异常值剔除功能的自监督损失进行训练，实现了高保真度、时间稳定且立体一致的 VR/AR 渲染效果，在用户研究中优于原始捕捉结果，且能泛化至未见过的主体。

ABSTRACT

Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real-time. We call this approach neural (re-)rendering, and our live system "LookinGood". Our deep architecture is trained to produce high resolution and high quality images from a coarse rendering in real-time. First, we propose a self-supervised training method that does not require manual ground-truth annotation. We contribute a specialized reconstruction error that uses semantic information to focus on relevant parts of the subject, e.g. the face. We also introduce a salient reweighing scheme of the loss function that is able to discard outliers. We specifically design the system for virtual and augmented reality headsets where the consistency between the left and right eye plays a crucial role in the final user experience. Finally, we generate temporally stable results by explicitly minimizing the difference between two consecutive frames. We tested the proposed system in two different scenarios: one involving a single RGB-D sensor, and upper body reconstruction of an actor, the second consisting of full body 360 degree capture. Through extensive experimentation, we demonstrate how our system generalizes across unseen sequences and subjects. The supplementary video is available at http://youtu.be/Md3tdAKoLGU.

研究动机与目标

• 解决 AR/VR 应用中实时动作捕捉所面临的视觉伪影问题，如孔洞、噪声、低分辨率和光照不良。
• 开发一种实时神经重渲染系统，无需人工标注即可增强来自 3D 重建的粗糙 2D 渲染结果。
• 确保时间稳定性与立体一致性，以优化 VR/AR 用户体验，特别是在双目渲染场景下。
• 通过使用语义感知损失函数的自监督训练，实现对未见主体和序列的良好泛化能力。

提出的方法

• 训练一个深度神经网络，实现实时将低分辨率、含伪影的 2D 渲染结果映射为高质量、高分辨率的输出。
• 采用自监督训练方法，通过重建损失结合语义监督，避免对真实标签的依赖。
• 在损失函数中引入基于显著性的重加权机制，突出关键区域（如面部）并抑制异常值。
• 通过模型量化（16 位浮点）和高效网络结构设计，优化实时推理性能，在 NVIDIA Titan V 上实现每对立体图像 29ms 的推理速度。
• 通过最小化帧间预测重渲染结果的差异，实现时间一致性。
• 系统在推理阶段还可预测一个二值分割掩码，将主体与背景分离。

实验结果

研究问题

• RQ1 深度学习模型能否在无人工标注的情况下，以自监督方式有效提升低质量实时动作捕捉输出？
• RQ2 如何在损失函数中利用语义信息，以提升面部等关键区域的视觉质量？
• RQ3 神经重渲染系统在 VR/AR 应用中能在多大程度上实现时间稳定性和立体一致性？
• RQ4 该系统在未见主体和序列上的泛化能力如何，尤其是在训练数据中未出现的情况下？

主要发现

• 系统在单张 GPU 上实现每对立体图像 29ms 的实时性能，满足 VR/AR 应用的需求。
• 在用户研究中，65% 的参与者更偏好神经重渲染结果而非真实标签，理由是其掩码更稳定，尽管分辨率略低。
• 模型对未见主体和序列具有良好的泛化能力，仅在输入严重损坏时出现渐进式退化。
• 结合显著性重加权的自监督损失显著提升了面部质量，并降低了异常值的影响。
• 通过最小化帧间重渲染输出的差异，系统实现了时间稳定性。
• 当输入严重损坏时，模型倾向于生成模糊的幻觉结果，表明在极端情况下存在关键局限性。