[Paper Review] Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos

Zhengxia Zou|arXiv (Cornell University)|Oct 22, 2020

Image Enhancement Techniques4 citations

TL;DR

This paper proposes a purely vision-based, real-time method for dynamic sky replacement and harmonization in videos, requiring no IMU or user interaction. It uses a coarse-to-fine sky matting network, motion estimation under affine sky motion, and a skybox for warping, blending, relighting, and color correction, achieving high-fidelity, visually realistic results across diverse handheld and dash-cam videos with controllable sky styles and dynamic lighting.

ABSTRACT

This paper proposes a vision-based method for video sky replacement and harmonization, which can automatically generate realistic and dramatic sky backgrounds in videos with controllable styles. Different from previous sky editing methods that either focus on static photos or require inertial measurement units integrated in smartphones on shooting videos, our method is purely vision-based, without any requirements on the capturing devices, and can be well applied to either online or offline processing scenarios. Our method runs in real-time and is free of user interactions. We decompose this artistic creation process into a couple of proxy tasks including sky matting, motion estimation, and image blending. Experiments are conducted on videos diversely captured in the wild by handheld smartphones and dash cameras, and show high fidelity and good generalization of our method in both visual quality and lighting/motion dynamics. Our code and animated results are available at \url{https://jiupinjia.github.io/skyar/}.

Motivation & Objective

• Address the lack of automatic, real-time sky editing solutions for dynamic videos, especially outside of professional film workflows.
• Overcome limitations of prior methods that rely on IMU data or are restricted to static images.
• Enable fully automatic, one-click sky replacement in both online augmented reality and offline video editing scenarios.
• Achieve visually realistic blending by harmonizing lighting, color, and motion dynamics between foreground and synthesized sky.
• Generalize across diverse real-world video conditions, including varying weather, time of day, and camera motion.

Proposed method

• Employ a deep learning-based, coarse-to-fine sky matting network that predicts soft sky mattes instead of binary segmentation for improved boundary accuracy.
• Use coordinate convolution (CoordConv) layers to inject spatial positional priors, enhancing matting performance on sky boundaries.
• Estimate sky motion using feature tracking under the assumption that sky and in-sky objects are at infinity, modeling their motion as affine relative to the foreground.
• Implement a skybox module that warps the selected sky template using estimated motion parameters and blends it with the foreground using linear blending.
• Apply two-step color correction: (1) color transfer to align foreground and sky color tones, and (2) relighting to adjust ambient illumination for realism.
• Integrate all components into an end-to-end, real-time pipeline suitable for both online and offline video processing.

Experimental results

Research questions

• RQ1 Can dynamic sky replacement in videos be achieved without relying on IMU or user interaction?
• RQ2 How can soft sky matting improve blending quality compared to binary segmentation in sky replacement tasks?
• RQ3 To what extent can vision-based motion estimation accurately model sky motion in real-world videos?
• RQ4 How effective are color transfer and relighting in achieving visual harmony between foreground and synthesized sky?
• RQ5 What are the failure modes of a purely vision-based sky replacement system under challenging conditions like nighttime or textureless skies?

Key findings

• The proposed two-stage soft matting network improves visual quality at sky boundaries compared to one-stage or hard segmentation methods, despite only a minor 0.14% PSNR gain.
• Removing coordinate convolution layers reduces PSNR by 0.26–0.30 and SSIM by 0.005–0.015, confirming their importance for spatial reasoning in sky matting.
• Color transfer and relighting significantly enhance realism: color transfer resolves tone mismatches, and relighting improves ambient light consistency.
• The method achieves high-fidelity results on diverse real-world videos from smartphones and dash cameras, with strong generalization across lighting and motion dynamics.
• Limitations include failure on nighttime videos (due to lack of training data) and inaccurate motion estimation when sky features are absent or sparse.
• The framework enables controllable weather and lighting transitions (e.g., day to night, cloudy to rainy), demonstrating utility beyond simple sky replacement.