[Paper Review] Hybrid Neural Fusion for Full-frame Video Stabilization

Yu-Lun Liu, Wei‐Sheng Lai+3 more|arXiv (Cornell University)|Feb 11, 2021

Image and Video Stabilization68 references4 citations

TL;DR

This paper proposes a learning-based hybrid neural fusion method for full-frame video stabilization that synthesizes stabilized frames by fusing warped features from multiple neighboring frames. By combining feature-level and image-level fusion with learned blending weights and residual detail transfer, the method reduces artifacts from optical flow inaccuracy and avoids aggressive cropping, achieving full-field-of-view stabilization with superior visual quality on NUS, Selfie, and DeepStab datasets.

ABSTRACT

Existing video stabilization methods often generate visible distortion or require aggressive cropping of frame boundaries, resulting in smaller field of views. In this work, we present a frame synthesis algorithm to achieve full-frame video stabilization. We first estimate dense warp fields from neighboring frames and then synthesize the stabilized frame by fusing the warped contents. Our core technical novelty lies in the learning-based hybrid-space fusion that alleviates artifacts caused by optical flow inaccuracy and fast-moving objects. We validate the effectiveness of our method on the NUS, selfie, and DeepStab video datasets. Extensive experiment results demonstrate the merits of our approach over prior video stabilization methods.

Motivation & Objective

• Address the limitation of existing video stabilization methods that require aggressive cropping due to missing pixels at frame borders.
• Overcome artifacts from optical flow inaccuracy and fast-moving objects in stabilized video frames.
• Enable full-field-of-view stabilization without resolution loss by fusing multiple neighboring frames.
• Improve visual quality through learned blending weights and high-frequency detail transfer.
• Maintain or expand the original field of view while ensuring temporal stability and reduced distortion.

Proposed method

• Estimate dense warp fields from neighboring frames using optical flow or motion estimation networks.
• Encode input frames into rich CNN-based feature representations for robust fusion.
• Implement a hybrid fusion mechanism combining feature-level and image-level fusion to reduce sensitivity to flow errors.
• Learn spatially varying blending weights to dynamically combine warped features and minimize artifacts.
• Reconstruct the final stabilized frame using a neural decoder network with residual detail transfer for sharper output.
• Apply path adjustment to balance motion smoothing and frame coverage, minimizing regions with no valid input.

Figure 2: Limitations of current state-of-the-art video stabilization techniques. (a) Current commercial video stabilization software (Adobe Premiere Pro 2020) fails to generate smooth videos in challenging scenarios of rapid camera shakes. (b) Yu and Ramamoorthi’s method [ 70 ] produces a temporall

Experimental results

Research questions

• RQ1 Can a learning-based fusion strategy effectively reduce artifacts caused by optical flow inaccuracy in video stabilization?
• RQ2 How can multiple warped frames be combined to achieve full-frame stabilization without aggressive cropping?
• RQ3 To what extent does hybrid feature and image-level fusion improve visual quality compared to single-level fusion?
• RQ4 Can the proposed method maintain or expand the original field of view while ensuring temporal stability?
• RQ5 How does the integration of learned blending weights and residual detail transfer enhance the sharpness and realism of stabilized frames?

Key findings

• The proposed method achieves a 100% cropping ratio on the NUS and Selfie datasets, indicating no frame border cropping and full-field-of-view stabilization.
• On the Selfie dataset, the method improves C (content), D (distortion), S (stability), and A (artifacts) scores to 1.00, 0.87, 0.87, and 0.64 respectively when combined with Yu and Ramamoorthi’s flow smoothing method.
• The method achieves the best LPIPS (0.12), SSIM (0.882), and PSNR (23.25) on the Trucks sequence of the Tanks and Temples dataset, outperforming FVS and other SOTA view synthesis methods.
• Visual comparisons show that the proposed method produces fewer artifacts and sharper results than DIFRINT and Adobe Premiere Pro, especially in high-motion and occlusion-prone scenes.
• Ablation studies confirm that hybrid fusion, learned blending weights, and residual detail transfer each contribute significantly to reducing blur and distortion.
• The method maintains strong performance across diverse datasets, including NUS (challenging handheld videos), Selfie (selfie videos with large motion), and DeepStab (high-motion stabilization tasks).

Figure 4: Learning-based fusion. Given the warped features $\{f_{n^{s}}^{k^{t}}\}_{n\in\Omega_{k}}$ (or warped images for image-space fusion), warping masks $\{M_{n^{s}}^{k^{t}}\}_{n\in\Omega_{k}}$ , and the flow error maps $\{e_{n^{s}}^{k^{t}}\}_{n\in\Omega_{k}}$ , we first concatenate feature, war