[Paper Review] Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation

Lingjie Liu, Weipeng Xu+6 more|arXiv (Cornell University)|Jan 14, 2020

Advanced Vision and ImagingComputer Science90 references49 citations

TL;DR

A three-stage neural pipeline disentangles pose-dependent fine-scale details in texture space from 2D screen-space embedding to synthesize temporally coherent, high-fidelity human videos. It uses TexNet to generate dynamic textures in UV space and RefNet to render and refine the final video.

ABSTRACT

Synthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.

Motivation & Objective

• Motivate improved realism in neural human video synthesis beyond 2D image translation artifacts like over-smoothing and temporal instability.
• Disentangle time-coherent fine-scale details from 2D pose embedding by leveraging texture-space learning.
• Develop a two-network system (TexNet and RefNet) to generate dynamic textures and refine rendered outputs.
• Enable applications such as motion transfer, interactive reenactment, and novel-view synthesis from monocular video.
• Provide a pipeline that preserves spatial, temporal, and geometric coherence of clothing and body appearance.

Proposed method

• Three-stage pipeline combining texture-space learning with image-space refinement.
• TexNet learns pose-dependent, time-coherent high-frequency texture details from a partial normal map in UV space.
• Partial dynamic textures are back-projected from monocular video frames using a performance capture mesh.
• A second network completes and renders the textured mesh to produce a coherent texture-space synthesis.
• RefNet refines the rendered texture-based mesh output to generate final photorealistic video including shadows and foreground-background interactions.
• Training uses cGAN-based objectives with frame and video losses and flow consistency.

Experimental results

Research questions

• RQ1 Can disentangling texture-space dynamics from screen-space embedding improve temporal coherence and detail in neural human video synthesis?
• RQ2 How can dynamic textures in UV space be learned from monocular data and applied to pose-driven rendering?
• RQ3 Does a two-network refinement (TexNet + RefNet) outperform prior 2D image-to-image translation approaches in motion transfer and novel-view synthesis?
• RQ4 What is the impact of using partial normal maps as pose encoding on the quality and stability of texture synthesis?

Key findings

• The three-stage approach yields temporally coherent, high-frequency details such as wrinkles that move with clothing.
• TexNet generates pose-dependent textures in UV space enabling accurate detail without per-frame 3D re-synthesis.
• RefNet effectively blends foreground/background, captures shadows, and corrects geometric errors, improving realism.
• The method enables motion transfer, interactive reenactment, and monocular novel-view synthesis with improved qualitative and quantitative results over state-of-the-art methods.
• Texture-space learning reduces 2D synthesis artifacts like missing limbs and silhouette errors common in prior frame-level image translations.