[Paper Review] Deep Motion Blind Video Stabilization

Muhammad Kashif Ali, Sangjoon Yu+1 more|arXiv (Cornell University)|Nov 19, 2020

Image and Video Stabilization4 citations

TL;DR

This paper proposes the first motion-blind, deep-learning-based video stabilization method that directly synthesizes stabilized frames without relying on explicit motion estimation. Using a novel unsupervised dataset generation pipeline based on iterative frame interpolation, the approach achieves full-frame stabilization with improved visual fidelity and ~3× speed-up over state-of-the-art methods, outperforming motion-aware baselines in distortion and cropping metrics while preserving resolution and content.

ABSTRACT

Despite the advances in the field of generative models in computer vision, video stabilization still lacks a pure regressive deep-learning-based formulation. Deep video stabilization is generally formulated with the help of explicit motion estimation modules due to the lack of a dataset containing pairs of videos with similar perspective but different motion. Therefore, the deep learning approaches for this task have difficulties in the pixel-level synthesis of latent stabilized frames, and resort to motion estimation modules for indirect transformations of the unstable frames to stabilized frames, leading to the loss of visual content near the frame boundaries. In this work, we aim to declutter this over-complicated formulation of video stabilization with the help of a novel dataset that contains pairs of training videos with similar perspective but different motion, and verify its effectiveness by successfully learning motion blind full-frame video stabilization through employing strictly conventional generative techniques and further improve the stability through a curriculum-learning inspired adversarial training strategy. Through extensive experimentation, we show the quantitative and qualitative advantages of the proposed approach to the state-of-the-art video stabilization approaches. Moreover, our method achieves $\sim3 imes$ speed-up over the currently available fastest video stabilization methods.

Motivation & Objective

• To address the lack of a pure regressive deep learning formulation for video stabilization due to insufficient paired training data with similar perspectives but different motions.
• To eliminate dependency on explicit motion estimation modules by enabling direct pixel-level synthesis of stabilized frames.
• To develop a scalable, unsupervised dataset generation pipeline that produces equi-perspective stable videos from unstable hand-held recordings.
• To improve temporal consistency and visual quality through a curriculum learning-inspired adversarial training strategy.
• To demonstrate that motion-blind stabilization is feasible and effective using conventional generative networks with proper data and training design.

Proposed method

• Proposes a dataset generation pipeline (DGP) that uses iterative frame interpolation to synthesize high-quality, equi-perspective stable videos from unstable hand-held videos.
• Employs a conventional U-Net-based generator network to directly predict stabilized frames from unstable inputs, bypassing optical flow or warping modules.
• Introduces a contrastive motion loss to encourage temporal consistency and reduce jitter in the stabilized output.
• Applies a curriculum learning strategy that progressively increases training complexity, allowing the network to learn stabilization in stages.
• Uses a temporal adversarial training strategy to enhance visual realism and temporal coherence of the stabilized video sequence.
• Leverages the generated dataset to train a motion-blind stabilization network that preserves full-frame resolution and visual content.

Experimental results

Research questions

• RQ1 Can full-frame video stabilization be achieved without explicit motion estimation, relying solely on end-to-end pixel-level synthesis?
• RQ2 Can a motion-blind deep learning model outperform motion-aware state-of-the-art methods in visual quality and stability?
• RQ3 Does a dataset with consistent perspective across unstable and stable video pairs enable better generalization and performance in video stabilization?
• RQ4 Can curriculum learning and adversarial training improve temporal consistency and visual fidelity in motion-blind stabilization?
• RQ5 Is it possible to achieve significant speed-up in video stabilization without sacrificing quality or content preservation?

Key findings

• The proposed method achieves ~3× speed-up over the currently fastest video stabilization methods, significantly improving inference efficiency.
• On the NUS dataset, the method outperforms state-of-the-art approaches in distortion and cropping metrics, indicating superior preservation of visual content and frame resolution.
• The model performs competitively in stability metrics on videos with motion profiles similar to those in the training data (e.g., Crowd, Parallax, Quick Rotation), with performance lagging slightly on other categories due to dataset bias.
• Visual comparisons show that the proposed method preserves scale and content better than Adobe Premiere 2018 CC, Robust L1, DIFRINT, and iterated CAIN, with fewer artifacts and higher resolution.
• The user study and supplementary results confirm that the proposed method produces more visually consistent and natural-looking stabilized videos.
• The ablation study confirms that the contrastive motion loss and curriculum learning strategy significantly improve temporal consistency and reduce flicker and jitter in the output.