[Paper Review] Video Frame Interpolation via Adaptive Separable Convolution

Simon Niklaus, Long Mai+1 more|arXiv (Cornell University)|Aug 5, 2017

Advanced Vision and ImagingComputer Science39 references72 citations

TL;DR

A neural network estimates dense pairs of 1D kernels per pixel to perform separable, spatially adaptive convolution for video frame interpolation, enabling full-frame synthesis with lower memory and the option to use perceptual loss for better visual quality.

ABSTRACT

Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.

Motivation & Objective

• Motivate end-to-end high-quality frame interpolation without explicit optical flow estimation.
• Reduce memory and computational demands of spatially adaptive kernels for large motion.
• Propose a fully convolutional network that predicts separable 1D kernels for all pixels simultaneously.
• Enable incorporation of perceptual losses to improve visual quality of interpolated frames.

Proposed method

• Replace full 2D adaptive kernels with separable 1D kernels to approximate 2D kernels for each output pixel.
• Use a fully convolutional encoder–decoder network to predict four 1D kernel sets per pixel (two frames, two directions).
• Apply the predicted 1D kernels as local convolutions over input frames to synthesize the intermediate frame in one pass.
• Train with L1 loss or perceptual loss (VGG-based feature reconstruction) to improve sharpness and detail.
• Handle boundary via replication padding and mitigate checkerboard artifacts by choosing bilinear upsampling in the decoder.
• Experiment with kernel size (51) and pooling layers (five) to balance motion handling and receptive field.

Experimental results

Research questions

• RQ1 Can separable 1D kernels approximate full 2D spatially adaptive kernels for frame interpolation while reducing memory requirements?
• RQ2 Does end-to-end training with perceptual loss yield higher perceptual quality for interpolated frames compared to purely pixel-wise losses?
• RQ3 How does the proposed separable convolution approach compare to state-of-the-art optical-flow-based and AdaConv methods in quality and speed?
• RQ4 What kernel size and network architecture choices best handle large motion and maintain full-frame synthesis at 1080p?
• RQ5 Is the method robust across challenging scenarios like occlusion, motion discontinuities, and brightness changes?

Key findings

• The separable 1D kernel approach reduces memory from n^2 to 2n per kernel, enabling full-frame 1080p interpolation in one pass.
• The L1 loss yields strong numerical performance and state-of-the-art results on Middlebury, particularly in regions with discontinuous motion.
• Incorporating perceptual loss (L_F) improves visual sharpness and high-frequency details, as shown in qualitative and user-study results.
• The method is significantly faster (over 20x) than AdaConv for 1080p interpolation and often produces visually more pleasing results.
• Using bilinear upsampling in the decoder helps mitigate checkerboard artifacts associated with some upsampling methods.
• Quantitative results show competitive MAE and SSIM with state-of-the-art methods, with the L1 model performing best overall on held evaluation.