[Paper Review] ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis

Zhouyong Liu, Shun Luo+6 more|arXiv (Cornell University)|Nov 20, 2020

Advanced Vision and Imaging47 references60 citations

TL;DR

ConvTransformer introduces a multi-head convolutional self-attention architecture that unifies video frame interpolation and extrapolation, achieving state-of-the-art-like results while enabling parallel training.

ABSTRACT

Deep Convolutional Neural Networks (CNNs) are powerful models that have achieved excellent performance on difficult computer vision tasks. Although CNNs perform well whenever large labeled training samples are available, they work badly on video frame synthesis due to objects deforming and moving, scene lighting changes, and cameras moving in video sequence. In this paper, we present a novel and general end-to-end architecture, called convolutional Transformer or ConvTransformer, for video frame sequence learning and video frame synthesis. The core ingredient of ConvTransformer is the proposed attention layer, i.e., multi-head convolutional self-attention layer, that learns the sequential dependence of video sequence. ConvTransformer uses an encoder, built upon multi-head convolutional self-attention layer, to encode the sequential dependence between the input frames, and then a decoder decodes the long-term dependence between the target synthesized frames and the input frames. Experiments on video future frame extrapolation task show ConvTransformer to be superior in quality while being more parallelizable to recent approaches built upon convolutional LSTM (ConvLSTM). To the best of our knowledge, this is the first time that ConvTransformer architecture is proposed and applied to video frame synthesis.

Motivation & Objective

• Motivate and address the challenge of video frame synthesis where objects move, deform, and lighting changes occur.
• Propose a unified end-to-end architecture that handles both interpolation and extrapolation.
• Develop a multi-head convolutional self-attention mechanism to model long-range dependencies across frames.
• Enable parallel training and inference to improve efficiency over recurrent architectures.

Proposed method

• Embed input frames into compact feature maps via a shared 4-layer CNN.
• Apply 3D positional encodings to preserve frame order information.
• Encode the frame sequence with stacked encoder layers using multi-head convolutional self-attention and convolutional feed-forward networks.
• Decode using a decoder that attends to encoded features and query frames, enabling learned long-range dependencies.
• Synthesize final frames with a 2-stage Synthesis Feed-Forward Network (SFFN) in a U-Net-like structure.
• Train with pixel-wise MSE loss to minimize reconstruction error between synthesized and ground-truth frames.

Experimental results

Research questions

• RQ1 Can ConvTransformer jointly handle video frame interpolation and extrapolation in a single, end-to-end architecture?
• RQ2 Does multi-head convolutional self-attention effectively capture long-range temporal and spatial dependencies in video sequences?
• RQ3 How does ConvTransformer perform relative to specialized interpolation and extrapolation methods on standard benchmarks?

Key findings

• ConvTransformer outperforms ConvLSTM-based extrapolation baselines (e.g., MCNet) on several benchmarks, especially for next-frame extrapolation.
• It achieves higher PSNR/SSIM on multiple datasets for both interpolation and extrapolation tasks compared with several state-of-the-art methods.
• The model shows favorable average performance across datasets, demonstrating the generality of the unified approach.
• Qualitative results indicate sharper, more photorealistic frames with fewer artifacts compared to prior methods.