[论文解读] Transformation-Based Models of Video Sequences

Joost R. van Amersfoort, Anitha Kannan+4 more|arXiv (Cornell University)|Jan 29, 2017

Advanced Image Processing Techniques参考文献 10被引用 63

一句话总结

本文提出通过对图像块预测局部仿射变换来预测下一个视频帧，从而在较小模型下实现清晰生成，并引入基于分类器的评估协议来评估生成帧。

ABSTRACT

In this work we propose a simple unsupervised approach for next frame prediction in video. Instead of directly predicting the pixels in a frame given past frames, we predict the transformations needed for generating the next frame in a sequence, given the transformations of the past frames. This leads to sharper results, while using a smaller prediction model. In order to enable a fair comparison between different video frame prediction models, we also propose a new evaluation protocol. We use generated frames as input to a classifier trained with ground truth sequences. This criterion guarantees that models scoring high are those producing sequences which preserve discriminative features, as opposed to merely penalizing any deviation, plausible or not, from the ground truth. Our proposed approach compares favourably against more sophisticated ones on the UCF-101 data set, while also being more efficient in terms of the number of parameters and computational cost.

研究动机与目标

• Motivate unsupervised learning for next-frame prediction in video.
• Propose a transformation-space approach to produce sharp, plausible frames with a compact model.
• Develop a patch-based affine transform extractor and a CNN predictor for next-frame transforms.
• Introduce a classifier-based evaluation protocol to assess generation quality beyond pixel-wise similarity.

提出的方法

• Tile frames into overlapping patches and estimate an affine transform for each patch to warp the input frame toward the next frame.
• Train a CNN that takes past affine transforms (from several consecutive frame pairs) and predicts the next set of affine transforms.
• Unroll the predictor in time to forecast multiple future frames and back-propagate through the unrolled network.
• Reconstruct predicted frames by applying the predicted affine transforms to the last observed frame and averaging overlapping predictions.
• Evaluate generations by feeding them to a pretrained classifier on ground-truth sequences to measure preservation of discriminative features.

实验结果

研究问题

• RQ1 Can motion in video be effectively modeled as local affine transformations applied to image patches?
• RQ2 Are patch-wise affine transformation predictions able to generate plausible future frames with less computational cost than pixel-based models?
• RQ3 Does a classifier-based evaluation protocol reliably reflect the quality of generated video sequences?
• RQ4 How does the transformation-based approach compare to optical-flow and adversarially trained pixel-based models on standard benchmarks?
• RQ5 Does unrolling the predictor over multiple steps improve multi-step prediction robustness?

主要发现

方法	4 帧	8 帧
真实帧	72.46	72.29
使用真实仿射变换	71.7	71.28
复制最后一帧	60.76	54.27
光流	57.29	49.37
Mathieu et al. (2016)	57.98	47.01
我们的方法 - 一步预测（未展开）	64.13	57.63
我们的方法 - 四步预测（展开4次）	64.54	57.88

• The transformation-space model yields sharper predictions and requires fewer parameters than competing models.
• On UCF-101, the affine-transform approach outperforms optical-flow baselines and the adversarial CNN in several settings while using fewer computations.
• Using ground-truth affine transforms yields competitive performance, validating the patch-wise affine decomposition.
• The unrolled multi-step predictor performs better than a greedy one-step predictor, indicating robustness to error accumulation.
• The best reported UCF-101 results (4-frame input, 8-frame prediction) show 64.54% with ground-truth frames and 57.88% with unrolled predictions, outperforming several baselines.
• The approach provides a strong, scalable baseline for next-frame prediction in natural videos.