[论文解读] ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis

Zhouyong Liu, Shun Luo+6 more|arXiv (Cornell University)|Nov 20, 2020

Advanced Vision and Imaging参考文献 47被引用 60

一句话总结

ConvTransformer 引入了多头卷积自注意力架构，将视频帧的插值与外推统一起来，在实现类似于最先进水平的结果的同时，实现并行训练。

ABSTRACT

Deep Convolutional Neural Networks (CNNs) are powerful models that have achieved excellent performance on difficult computer vision tasks. Although CNNs perform well whenever large labeled training samples are available, they work badly on video frame synthesis due to objects deforming and moving, scene lighting changes, and cameras moving in video sequence. In this paper, we present a novel and general end-to-end architecture, called convolutional Transformer or ConvTransformer, for video frame sequence learning and video frame synthesis. The core ingredient of ConvTransformer is the proposed attention layer, i.e., multi-head convolutional self-attention layer, that learns the sequential dependence of video sequence. ConvTransformer uses an encoder, built upon multi-head convolutional self-attention layer, to encode the sequential dependence between the input frames, and then a decoder decodes the long-term dependence between the target synthesized frames and the input frames. Experiments on video future frame extrapolation task show ConvTransformer to be superior in quality while being more parallelizable to recent approaches built upon convolutional LSTM (ConvLSTM). To the best of our knowledge, this is the first time that ConvTransformer architecture is proposed and applied to video frame synthesis.

研究动机与目标

• Motivate and address the challenge of video frame synthesis where objects move, deform, and lighting changes occur.
• Propose a unified end-to-end architecture that handles both interpolation and extrapolation.
• Develop a multi-head convolutional self-attention mechanism to model long-range dependencies across frames.
• Enable parallel training and inference to improve efficiency over recurrent architectures.

提出的方法

• Embed input frames into compact feature maps via a shared 4-layer CNN.
• Apply 3D positional encodings to preserve frame order information.
• Encode the frame sequence with stacked encoder layers using multi-head convolutional self-attention and convolutional feed-forward networks.
• Decode using a decoder that attends to encoded features and query frames, enabling learned long-range dependencies.
• Synthesize final frames with a 2-stage Synthesis Feed-Forward Network (SFFN) in a U-Net-like structure.
• Train with pixel-wise MSE loss to minimize reconstruction error between synthesized and ground-truth frames.

实验结果

研究问题

• RQ1 Can ConvTransformer jointly handle video frame interpolation and extrapolation in a single, end-to-end architecture?
• RQ2 Does multi-head convolutional self-attention effectively capture long-range temporal and spatial dependencies in video sequences?
• RQ3 How does ConvTransformer perform relative to specialized interpolation and extrapolation methods on standard benchmarks?

主要发现

• ConvTransformer outperforms ConvLSTM-based extrapolation baselines (e.g., MCNet) on several benchmarks, especially for next-frame extrapolation.
• It achieves higher PSNR/SSIM on multiple datasets for both interpolation and extrapolation tasks compared with several state-of-the-art methods.
• The model shows favorable average performance across datasets, demonstrating the generality of the unified approach.
• Qualitative results indicate sharper, more photorealistic frames with fewer artifacts compared to prior methods.