[Paper Review] Decoupled Spatial-Temporal Transformer for Video Inpainting

Rui Liu, Hanming Deng+7 more|arXiv (Cornell University)|Apr 14, 2021

Generative Adversarial Networks and Image SynthesisComputer Science34 references47 citations

TL;DR

Introduces a Decoupled Spatial-Temporal Transformer (DSTT) with a hierarchical encoder to efficiently perform video inpainting by separately modeling temporal object movement and background texture propagation, achieving state-of-the-art results with higher efficiency.

ABSTRACT

Video inpainting aims to fill the given spatiotemporal holes with realistic appearance but is still a challenging task even with prosperous deep learning approaches. Recent works introduce the promising Transformer architecture into deep video inpainting and achieve better performance. However, it still suffers from synthesizing blurry texture as well as huge computational cost. Towards this end, we propose a novel Decoupled Spatial-Temporal Transformer (DSTT) for improving video inpainting with exceptional efficiency. Our proposed DSTT disentangles the task of learning spatial-temporal attention into 2 sub-tasks: one is for attending temporal object movements on different frames at same spatial locations, which is achieved by temporally-decoupled Transformer block, and the other is for attending similar background textures on same frame of all spatial positions, which is achieved by spatially-decoupled Transformer block. The interweaving stack of such two blocks makes our proposed model attend background textures and moving objects more precisely, and thus the attended plausible and temporally-coherent appearance can be propagated to fill the holes. In addition, a hierarchical encoder is adopted before the stack of Transformer blocks, for learning robust and hierarchical features that maintain multi-level local spatial structure, resulting in the more representative token vectors. Seamless combination of these two novel designs forms a better spatial-temporal attention scheme and our proposed model achieves better performance than state-of-the-art video inpainting approaches with significant boosted efficiency.

Motivation & Objective

• Motivate robust and efficient video inpainting that preserves temporal coherence and texture quality.
• Develop a Transformer-based framework that decouples spatial and temporal attention to reduce computation.
• Incorporate a hierarchical encoder to preserve multi-level local spatial structure before attention-based propagation.
• Demonstrate state-of-the-art performance on video completion and object removal with improved efficiency.

Proposed method

• Propose a hierarchical encoder that mixes multi-level features frame-by-frame to produce robust token representations.
• Use an interweaving stack of two Transformer blocks: temporally-decoupled attention across the same spatial zones over time, and spatially-decoupled attention within each frame across all tokens.
• Split feature maps into s×s zones to make temporal and spatial attention easier and more efficient.
• Stack blocks in an alternating TDB-SDB pattern to enable thorough spatiotemporal propagation.
• Train end-to-end in a self-supervised setting by masking videos and reconstructing them with reconstruction and adversarial losses (L_hole, L_valid, L_adv).
• Compose G = G_HE ∘ G_DSTT ∘ G_Dec, where G_HE is the hierarchical encoder, G_DSTT the decoupled Transformer stack, and G_Dec the CNN decoder.

Experimental results

Research questions

• RQ1 Can decoupling spatial and temporal attention reduce computational cost while maintaining or improving inpainting quality?
• RQ2 Does a hierarchical encoder improve the representational quality of tokens for Transformer-based video inpainting?
• RQ3 How does the interweaving pattern of temporally-decoupled and spatially-decoupled blocks affect reconstruction quality and efficiency?
• RQ4 What are the gains in PSNR/SSIM/VFID and speed compared to state-of-the-art methods on standard video datasets?
• RQ5 Is the method robust for both video completion and object removal tasks?

Key findings

• DSTT achieves state-of-the-art PSNR and competitive SSIM and VFID on YouTube-VOS and DAVIS for video completion.
• The proposed method attains the highest FPS (37.3) with relatively low FLOPs (128B) among reported methods.
• Ablation shows hierarchical encoder improves PSNR by up to ~0.4 points and reduces VFID, confirming the benefit of multi-level feature mixing.
• Splitting feature maps into zones (s=2) yields better accuracy-efficiency trade-offs than a single global attention (s=1) or too fine zoning (s>4).
• Interweaving temporally-decoupled and spatially-decoupled blocks consistently outperforms single-block variants, highlighting the benefit of alternate spatiotemporal propagation.