[Paper Review] Learning Joint Spatial-Temporal Transformations for Video Inpainting

Yanhong Zeng, Jianlong Fu+1 more|arXiv (Cornell University)|Jul 20, 2020

Generative Adversarial Networks and Image SynthesisComputer Science23 references18 citations

TL;DR

This paper proposes STTN, a joint spatial-temporal transformer network for video inpainting that simultaneously completes missing regions across all input frames using multi-scale patch-based self-attention. By optimizing with a spatial-temporal adversarial loss, STTN achieves state-of-the-art performance, improving PSNR by 2.4% and VFID by 19.7% over prior methods on challenging masks.

ABSTRACT

High-quality video inpainting that completes missing regions in video frames is a promising yet challenging task. State-of-the-art approaches adopt attention models to complete a frame by searching missing contents from reference frames, and further complete whole videos frame by frame. However, these approaches can suffer from inconsistent attention results along spatial and temporal dimensions, which often leads to blurriness and temporal artifacts in videos. In this paper, we propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. Specifically, we simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss. To show the superiority of the proposed model, we conduct both quantitative and qualitative evaluations by using standard stationary masks and more realistic moving object masks. Demo videos are available at https://github.com/researchmm/STTN.

Motivation & Objective

• To address temporal inconsistency and blurriness in frame-by-frame video inpainting by modeling spatial and temporal dependencies jointly.
• To overcome limitations of frame-wise or step-by-step attention mechanisms that fail under complex motion and appearance changes.
• To improve video inpainting quality by enabling simultaneous completion of all input frames using a unified transformer architecture.
• To enhance perceptual quality and temporal coherence through a novel spatial-temporal adversarial loss.
• To enable fast training and inference via multi-scale, non-overlapping patch representations in the transformer.

Proposed method

• Proposes a multi-scale patch-based attention module that extracts non-overlapping patches of varying sizes from all frames to capture diverse motion patterns.
• Employs a multi-head transformer to compute similarity across spatial patches at different scales, aggregating attention results for coherent content generation.
• Stacks multiple spatial-temporal transformer layers to iteratively refine missing region features using updated context.
• Introduces a spatial-temporal adversarial loss that jointly optimizes perceptual quality and temporal consistency across frames.
• Treats video inpainting as a 'multi-to-multi' task, where all input frames are processed simultaneously rather than sequentially.
• Uses a generator network with the STTN architecture to produce completed frames, trained end-to-end with the adversarial loss.

Experimental results

Research questions

• RQ1 Can a joint spatial-temporal transformer architecture improve temporal coherence and reduce blurriness in video inpainting compared to frame-by-frame methods?
• RQ2 How effective is multi-scale patch-based attention in modeling complex motions and appearance variations across frames?
• RQ3 Does stacking multiple transformer layers lead to consistent performance gains in video inpainting?
• RQ4 Can a spatial-temporal adversarial loss enhance perceptual quality and reduce temporal artifacts in generated videos?
• RQ5 How does the proposed method perform under realistic, free-form masks such as moving objects or large occlusions?

Key findings

• STTN achieves a 2.4% relative improvement in PSNR and a 19.7% relative improvement in VFID over state-of-the-art methods on DAVIS and YouTube-VOS datasets with free-form masks.
• The ablation study confirms that multi-scale patch representation significantly improves performance, with the best results achieved when combining patch sizes of 108×60, 36×20, 18×10, and 9×5.
• Stacking eight transformer layers yields the best performance, showing continuous improvement with deeper stacks, indicating the benefit of iterative refinement.
• User studies rank STTN as the best-performing model in 80% of trials for both stationary and moving masks, confirming superior perceptual quality.
• Despite strong results, STTN fails to capture rapid, continuous motions (e.g., dancing) under large masks, producing blurs due to limited 2D patch attention without 3D temporal modeling.
• The model demonstrates fast training and inference due to the efficient, non-overlapping multi-scale patch representation, suitable for real-time video understanding tasks.