[Paper Review] Natural & Adversarial Bokeh Rendering via Circle-of-Confusion Predictive Network

Yihao Huang, Felix Juefei-Xu+3 more|arXiv (Cornell University)|Nov 25, 2021

Advanced Image Processing Techniques50 references4 citations

TL;DR

This paper proposes CoCNet, a hybrid deep learning framework that predicts circle-of-confusion (CoC) parameters from all-in-focus and depth images to generate realistic bokeh effects using a physics-based rendering model. By combining data-driven learning with physical priors, CoCNet achieves state-of-the-art bokeh quality with a lightweight network and enables adversarial bokeh attacks that significantly degrade deep neural network performance while preserving visual naturalness.

ABSTRACT

Bokeh effect is a natural shallow depth-of-field phenomenon that blurs the out-of-focus part in photography. In recent years, a series of works have proposed automatic and realistic bokeh rendering methods for artistic and aesthetic purposes. They usually employ cutting-edge data-driven deep generative networks with complex training strategies and network architectures. However, these works neglect that the bokeh effect, as a real phenomenon, can inevitably affect the subsequent visual intelligent tasks like recognition, and their data-driven nature prevents them from studying the influence of bokeh-related physical parameters (i.e., depth-of-the-field) on the intelligent tasks. To fill this gap, we study a totally new problem, i.e., natural & adversarial bokeh rendering, which consists of two objectives: rendering realistic and natural bokeh and fooling the visual perception models (i.e., bokeh-based adversarial attack). To this end, beyond the pure data-driven solution, we propose a hybrid alternative by taking the respective advantages of data-driven and physical-aware methods. Specifically, we propose the circle-of-confusion predictive network (CoCNet) by taking the all-in-focus image and depth image as inputs to estimate circle-of-confusion parameters for each pixel, which are employed to render the final image through a well-known physical model of bokeh. With the hybrid solution, our method could achieve more realistic rendering results with the naive training strategy and a much lighter network.

Motivation & Objective

• To address the lack of physical realism and interpretability in purely data-driven bokeh rendering methods.
• To study the impact of bokeh-related physical parameters—especially depth-of-field—on visual intelligence tasks.
• To develop a hybrid approach that combines data-driven learning with physical priors for more controllable and interpretable bokeh generation.
• To enable adversarial bokeh attacks that exploit depth variations to degrade deep neural network performance.
• To explore the dual use of adversarial bokeh examples as data augmentation to improve downstream tasks like defocus deblurring.

Proposed method

• CoCNet takes an all-in-focus image and a depth map as inputs to predict per-pixel circle-of-confusion (CoC) parameters.
• The network uses a lightweight architecture to predict specialized convolution kernels that simulate physical blurring via CoC-based filtering.
• Final bokeh images are rendered by fusing the all-in-focus image with CoC-filtered templates using a known physical model of bokeh.
• Adversarial bokeh attacks are implemented by optimizing the depth map (while keeping CoCNet fixed) to maximize misclassification under PGD-based attacks.
• The method employs gradient-based, smooth gradient-based, and background-guided gradient-based attack strategies to generate natural-looking adversarial examples.
• The framework is fine-tuned on defocus deblurring datasets to evaluate the utility of adversarial bokeh examples as data augmentation.

Experimental results

Research questions

• RQ1 How can we generate more realistic and physically plausible bokeh effects by integrating physical priors into deep learning models?
• RQ2 To what extent do bokeh effects, particularly variations in depth-of-field, degrade the performance of visual perception models?
• RQ3 Can we generate adversarial bokeh examples that are visually natural yet highly effective in fooling deep neural networks?
• RQ4 How effective are adversarial bokeh examples as data augmentation for improving downstream tasks such as defocus deblurring?
• RQ5 What is the transferability of adversarial bokeh attacks across different deep neural network architectures?

Key findings

• CoCNet achieves comparable bokeh quality to state-of-the-art methods like PyNET and DMSHN while using less than 1/5 of the parameters and requiring significantly less inference time.
• The adversarial bokeh attack reduces top-1 accuracy of ResNet50, VGG, MobileNetV2, and DenseNet to below 10% on all tested models, demonstrating high attack success.
• Among attack variants, background-guided gradient-based attack (bg-gda) shows the best transferability across models, outperforming standard gradient and smooth gradient methods.
• The smooth gradient-based attack (sm-gda) produces more natural-looking adversarial examples than unrestricted attacks like ColorFool and ACE, reducing suspicion.
• Fine-tuning the IFAN model with adversarial bokeh examples leads to improved deblurring performance, with F-IFAN achieving slightly higher PSNR (25.38111 vs. 25.36620) and SSIM (0.78887 vs. 0.78885) on the DPDD test set.
• Failure cases show that all methods, including CoCNet, struggle with rendering small bokeh spots, indicating a need for future improvement in high-frequency bokeh detail modeling.