[Paper Review] Synthetic Defocus and Look-Ahead Autofocus for Casual Videography

Xuaner Zhang, Kevin Matzen+4 more|arXiv (Cornell University)|May 15, 2019

Image Processing Techniques and Applications61 references4 citations

TL;DR

This paper introduces RVR-LAAF, a system that enables cinematic shallow depth of field and context-aware autofocus in casual videography by synthesizing refocusable video from smartphone-recorded deep DOF footage and using AI-powered look-ahead analysis of future frames to anticipate focus transitions. The key contribution is a novel framework that overcomes real-time autofocus limitations by enabling future-aware focus decisions, achieving smooth, accurate subject tracking even during unpredictable actions.

ABSTRACT

In cinema, large camera lenses create beautiful shallow depth of field (DOF), but make focusing difficult and expensive. Accurate cinema focus usually relies on a script and a person to control focus in realtime. Casual videographers often crave cinematic focus, but fail to achieve it. We either sacrifice shallow DOF, as in smartphone videos; or we struggle to deliver accurate focus, as in videos from larger cameras. This paper is about a new approach in the pursuit of cinematic focus for casual videography. We present a system that synthetically renders refocusable video from a deep DOF video shot with a smartphone, and analyzes future video frames to deliver context-aware autofocus for the current frame. To create refocusable video, we extend recent machine learning methods designed for still photography, contributing a new dataset for machine training, a rendering model better suited to cinema focus, and a filtering solution for temporal coherence. To choose focus accurately for each frame, we demonstrate autofocus that looks at upcoming video frames and applies AI-assist modules such as motion, face, audio and saliency detection. We also show that autofocus benefits from machine learning and a large-scale video dataset with focus annotation, where we use our RVR-LAAF GUI to create this sizable dataset efficiently. We deliver, for example, a shallow DOF video where the autofocus transitions onto each person before she begins to speak. This is impossible for conventional camera autofocus because it would require seeing into the future.

Motivation & Objective

• To address the fundamental limitation of real-time camera autofocus in casual videography, where focus errors occur due to unpredictable subject motion and lack of script-based guidance.
• To enable shallow depth of field in smartphone videos, which traditionally sacrifice cinematic bokeh due to small sensor size and fixed focus.
• To overcome the inability of conventional autofocus to anticipate decisive actions, such as a person beginning to speak, by enabling 'future-aware' focus decisions.
• To develop a practical, end-to-end system that combines machine learning, physically-based rendering, and temporal coherence for synthetic refocusable video.
• To demonstrate that AI-assisted, look-ahead autofocus significantly improves focus accuracy and transition smoothness compared to real-time camera systems.

Proposed method

• Synthetic refocusable video is generated using a deep learning model trained on a new dataset of over 2,000 image pairs/triplets with varying aperture and focus settings.
• The system employs a physically-based rendering model that combines predicted depth, HDR estimation, and lens-specific blur kernels to simulate shallow DOF with cinematic bokeh.
• Temporal coherence is enforced via a filtering solution that stabilizes blur transitions across frames, reducing flicker and inconsistency.
• Look-Ahead Autofocus (LAAF) analyzes upcoming video frames using AI modules for motion, face detection, audio activity, and saliency to predict focus targets.
• The LAAF framework uses a GUI (RVR-LAAF) to efficiently annotate a large-scale video dataset with focus transitions for training.
• Focus decisions are made based on predictive analysis of future frames, enabling transitions before a subject speaks or acts, which is impossible for real-time camera autofocus.

Experimental results

Research questions

• RQ1 Can synthetic refocusable video be reliably generated from deep DOF smartphone video using machine learning and physically-based rendering?
• RQ2 Can future-frame analysis enable more accurate and anticipatory autofocus decisions than real-time camera systems?
• RQ3 How effective is AI-assisted look-ahead autofocus in tracking dynamic subjects and transitioning focus before key narrative actions?
• RQ4 To what extent does temporal filtering improve visual consistency in synthetic refocusable video?
• RQ5 Can the system simulate cinematic bokeh effects, such as anamorphic lens blur, using learned depth and HDR maps?

Key findings

• The RVR-LAAF system successfully generates refocusable video from smartphone-recorded deep DOF footage with improved visual quality and temporal stability through learned depth and HDR estimation.
• Look-Ahead Autofocus enables focus transitions onto a speaker before they begin speaking, a capability impossible for conventional real-time autofocus systems.
• The system demonstrates superior focus tracking on rapidly moving subjects, such as a soccer player, by anticipating actions and transitioning focus in advance.
• The method achieves accurate simulation of cinematic bokeh, including non-circular defocus blur, by modeling lens-specific blur kernels from professional cinematography lenses.
• Compared to high-end DSLR cameras, RVR-LAAF outperforms in focus transition accuracy during rapid subject changes, as validated in side-by-side video comparisons.
• The system reduces temporal flicker in defocus blur through a dedicated temporal stabilization module, significantly improving visual coherence in synthetic video.