[Paper Review] Self-Supervised Motion Magnification by Backpropagating Through Optical Flow

Zhaoying Pan, Daniel Geng+1 more|arXiv (Cornell University)|Nov 28, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a self-supervised motion magnification method that magnifies subtle video motions by backpropagating through a pretrained optical flow network to scale predicted flow by a desired factor. It trains without synthetic data using only real unlabeled videos, achieves state-of-the-art results on real and synthetic videos, and enables test-time adaptation and object-specific magnification via user-provided masks.

ABSTRACT

This paper presents a simple, self-supervised method for magnifying subtle motions in video: given an input video and a magnification factor, we manipulate the video such that its new optical flow is scaled by the desired amount. To train our model, we propose a loss function that estimates the optical flow of the generated video and penalizes how far if deviates from the given magnification factor. Thus, training involves differentiating through a pretrained optical flow network. Since our model is self-supervised, we can further improve its performance through test-time adaptation, by finetuning it on the input video. It can also be easily extended to magnify the motions of only user-selected objects. Our approach avoids the need for synthetic magnification datasets that have been used to train prior learning-based approaches. Instead, it leverages the existing capabilities of off-the-shelf motion estimators. We demonstrate the effectiveness of our method through evaluations of both visual quality and quantitative metrics on a range of real-world and synthetic videos, and we show our method works for both supervised and unsupervised optical flow methods.

Motivation & Objective

• To address the limitation of prior learning-based motion magnification methods that require large-scale synthetic datasets for supervision.
• To leverage existing off-the-shelf optical flow networks as a differentiable supervision signal, avoiding the need for custom training data.
• To enable self-supervised training on real, unlabeled videos by formulating a loss that penalizes deviation from the target magnification factor in predicted optical flow.
• To support test-time adaptation for improved generation quality and user-guided object-specific magnification using segmentation masks.

Proposed method

• The model takes a video frame pair and a magnification factor α as input, and generates a new frame pair with optical flow scaled by α.
• A differentiable loss function estimates the optical flow of the generated video and penalizes deviation from α times the input flow, using a pretrained optical flow network.
• The training process backpropagates gradients through the optical flow network to optimize the image generation network.
• A regularization loss preserves visual appearance by encouraging pixel-wise consistency between input and generated frames.
• Test-time adaptation is enabled by fine-tuning the model on the input video during inference to improve quality.
• Object-specific magnification is achieved by applying a user-provided segmentation mask to restrict flow scaling to specific regions.

Experimental results

Research questions

• RQ1 Can motion magnification be effectively trained in a self-supervised manner without synthetic data or ground-truth magnified pairs?
• RQ2 Can off-the-shelf optical flow networks serve as a reliable, differentiable supervision signal for motion magnification?
• RQ3 How does the method perform compared to supervised baselines and Lagrangian warping methods in terms of motion accuracy and visual quality?
• RQ4 To what extent can test-time adaptation improve the quality of magnified videos?
• RQ5 Can the method be extended to magnify motion in only user-selected objects using segmentation masks?

Key findings

• The proposed method outperforms baseline warping-based methods and supervised learning approaches in both quantitative metrics and visual quality, especially in complex scenes with occlusions.
• On synthetic videos, the method achieved a motion error of 0.61px (warp nearest) and 0.57px (warp bilinear) with nonparametric inpainting, significantly lower than baselines.
• With DeepFillv2 inpainting, the method achieved a magnification error of 0.66x (subpixel test) and 0.53x (noise test), outperforming all baseline methods.
• Test-time adaptation reduced motion error from 0.61px to 0.53px (warp nearest) and from 0.57px to 0.53px (warp bilinear), demonstrating consistent quality improvement.
• The method successfully magnified motions in thin structures like guitar strings and tree branches in some cases, though it occasionally failed due to flow estimation errors in such regions.
• Failure cases were primarily due to incorrect optical flow predictions in smooth regions or thin structures, which were then magnified, indicating sensitivity to flow network accuracy.