[Paper Review] Compressing Video Calls using Synthetic Talking Heads

Madhav Agarwal, Anchit Gupta+3 more|arXiv (Cornell University)|Oct 7, 2022

Speech and Audio Processing4 citations

TL;DR

This paper proposes an end-to-end video compression system for talking head video calls using synthetic talking heads, transmitting only key points and pivot frames to drastically reduce bandwidth. By leveraging face reenactment, dense optical flow, frame interpolation, and super-resolution, the method achieves a record-low 0.0012 BPP while maintaining high visual quality, with adaptive pivot frame selection enabling stable performance in long calls.

ABSTRACT

We leverage the modern advancements in talking head generation to propose an end-to-end system for talking head video compression. Our algorithm transmits pivot frames intermittently while the rest of the talking head video is generated by animating them. We use a state-of-the-art face reenactment network to detect key points in the non-pivot frames and transmit them to the receiver. A dense flow is then calculated to warp a pivot frame to reconstruct the non-pivot ones. Transmitting key points instead of full frames leads to significant compression. We propose a novel algorithm to adaptively select the best-suited pivot frames at regular intervals to provide a smooth experience. We also propose a frame-interpolater at the receiver's end to improve the compression levels further. Finally, a face enhancement network improves reconstruction quality, significantly improving several aspects like the sharpness of the generations. We evaluate our method both qualitatively and quantitatively on benchmark datasets and compare it with multiple compression techniques. We release a demo video and additional information at https://cvit.iiit.ac.in/research/projects/cvit-projects/talking-video-compression.

Motivation & Objective

• To address the growing bandwidth demands of video calling in low-connectivity regions by developing a specialized compression technique for talking head videos.
• To exploit high-level semantic content—such as facial landmarks, head pose, and expressions—for extreme video compression beyond generic codecs.
• To reduce bits-per-pixel (BPP) significantly while preserving perceptual quality in real-time video calls.
• To enable long-duration video calls with stable visual quality through adaptive pivot frame selection based on pose and background changes.
• To improve reconstruction quality using frame interpolation and super-resolution networks at the receiver end.

Proposed method

• Transmit a high-resolution pivot frame at the start of the call, followed by only 8-byte key points per non-pivot frame to represent facial geometry and motion.
• Use a state-of-the-art face reenactment network to detect and transmit key points, which are then used to compute dense optical flow for warping the pivot frame to reconstruct intermediate frames.
• Apply a novel frame-interpolater network at the receiver to generate missing frames between key-point-transmitted frames, reducing the need for frequent pivot updates.
• Employ a patch-wise super-resolution network to upsample reconstructed frames to arbitrary high resolutions, enhancing sharpness and visual fidelity.
• Introduce an adaptive pivot frame selection algorithm that triggers new pivot frames when head pose changes exceed 15° or background changes exceed a threshold (d_bg > 0.05), improving long-term reconstruction quality.
• Optimize key point representation from 24 to 8 bytes per point, significantly reducing transmission overhead.

Experimental results

Research questions

• RQ1 Can a talking head video compression system based on synthetic generation achieve significantly lower BPP than traditional codecs like H.264 and H.265?
• RQ2 How effective is using compact key points (8 bytes each) combined with dense flow warping for reconstructing high-quality talking head videos?
• RQ3 To what extent does frame interpolation at the receiver improve compression efficiency without increasing bandwidth?
• RQ4 How does adaptive pivot frame selection mitigate quality degradation in long video calls with dynamic head poses or backgrounds?
• RQ5 Can super-resolution reconstruction enhance visual quality sufficiently to compensate for low-bitrate transmission?

Key findings

• The proposed method achieves a BPP of 0.0012, which is significantly lower than H.264 (0.0057) and H.265 (0.0039), while maintaining competitive PSNR (26.73) and SSIM (0.81).
• Adding frame interpolation reduces BPP to 0.0006 while maintaining PSNR above 27.73, demonstrating effective compression with minimal quality loss.
• The super-resolution network improves PSNR from 25.34 (16×16 patches) to 27.47 (64×64 patches), showing that patch size selection impacts reconstruction quality.
• Adaptive pivot frame selection with thresholds γ > 15° and d_bg > 0.05 results in a pivot frame update every ~10 seconds, improving visual quality in long calls without excessive bandwidth cost.
• The method reduces key point transmission size from 24 to 8 bytes per point, enabling a 66% reduction in metadata overhead.
• Qualitative results show sharper, more natural-looking talking heads compared to baseline methods, especially in long-duration and dynamic video sequences.