[Paper Review] Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration

Liyuan Pan, S. A. H. Chowdhury+4 more|arXiv (Cornell University)|Dec 1, 2020

Image Processing Techniques and Applications38 references4 citations

TL;DR

This paper proposes DDDNet, an end-to-end deep learning framework that jointly estimates inverse depth maps and restores sharp images from dual-pixel (DP) image pairs by modeling the physical imaging process of DP sensors. It introduces a reblur loss derived from a mathematical DP model to regularize depth estimation, enabling superior performance in both depth estimation and deblurring, validated on synthetic and real datasets with state-of-the-art results.

ABSTRACT

The dual-pixel (DP) hardware works by splitting each pixel in half and creating an image pair in a single snapshot. Several works estimate depth/inverse depth by treating the DP pair as a stereo pair. However, dual-pixel disparity only occurs in image regions with the defocus blur. The heavy defocus blur in DP pairs affects the performance of matching-based depth estimation approaches. Instead of removing the blur effect blindly, we study the formation of the DP pair which links the blur and the depth information. In this paper, we propose a mathematical DP model which can benefit depth estimation by the blur. These explorations motivate us to propose an end-to-end DDDNet (DP-based Depth and Deblur Network) to jointly estimate the depth and restore the image. Moreover, we define a reblur loss, which reflects the relationship of the DP image formation process with depth information, to regularise our depth estimate in training. To meet the requirement of a large amount of data for learning, we propose the first DP image simulator which allows us to create datasets with DP pairs from any existing RGBD dataset. As a side contribution, we collect a real dataset for further research. Extensive experimental evaluation on both synthetic and real datasets shows that our approach achieves competitive performance compared to state-of-the-art approaches.

Motivation & Objective

• To address the challenge of low baseline and heavy defocus blur in dual-pixel (DP) image pairs, which degrade stereo matching and depth estimation performance.
• To exploit the intrinsic relationship between defocus blur, depth, and image formation in DP sensors, rather than treating blur as noise.
• To develop an end-to-end deep network that jointly estimates inverse depth and restores sharp images from a single DP snapshot.
• To create a differentiable DP image simulator capable of generating realistic DP pairs from any RGB-D dataset for large-scale training.
• To collect a real-world DP dataset to support further research in depth estimation and image restoration.

Proposed method

• Formulate a mathematical DP model that explicitly links depth, defocus blur, and the all-in-focus image through the imaging process of dual-pixel sensors.
• Design DDDNet, a joint encoder-decoder network that takes a DP image pair as input and outputs a deblurred image and an inverse depth map in an end-to-end manner.
• Introduce a reblur loss based on the DP model, which reconstructs the input DP images from the estimated depth and deblurred image to regularize depth prediction during training.
• Develop a differentiable DP image simulator that generates synthetic DP pairs from any existing RGB-D dataset by applying depth-dependent blur and sub-aperture sampling.
• Train DDDNet using synthetic data from the simulator and fine-tune on real data, leveraging the reblur loss for self-supervised adaptation.
• Use the reblur loss as a self-supervision signal to enable domain generalization and fine-tuning without ground-truth depth.

Experimental results

Research questions

• RQ1 Can the physical imaging model of dual-pixel sensors be leveraged to improve depth estimation and image deblurring simultaneously?
• RQ2 How can the defocus blur in DP pairs be used as a signal rather than a noise source in depth estimation?
• RQ3 Can a differentiable DP image simulator generate realistic training data that enables generalization to real-world scenarios?
• RQ4 Does integrating a reblur loss based on the DP model lead to more accurate depth and deblurring predictions?
• RQ5 Can the proposed method generalize to real datasets without fine-tuning, and how effective is self-supervised fine-tuning using the reblur loss?

Key findings

• On the synthetic DPD-blur dataset, DDDNet with the reblur loss achieves a PSNR of 26.92 dB, SSIM of 0.864, and RMSE_rel of 4.51% after fine-tuning, outperforming models trained directly on the same data.
• The reblur loss improves deblurring performance by approximately 10% relatively, as shown by a drop in RMSE_rel from 0.091 to 0.083 when using the loss in the DeblurNet.
• The model trained on the synthetic DP simulator generalizes well to real data: after fine-tuning on only half the DPD-blur dataset, it achieves PSNR of 26.52 dB and SSIM of 0.822 in 20 epochs.
• On the DPD-disp dataset, the model achieves second-best performance without fine-tuning, and further improves with self-supervised fine-tuning using the reblur loss, outperforming several state-of-the-art methods.
• Qualitative results in Figure 8 show that DDDNet produces sharper deblurred images and more accurate inverse depth maps than baselines like DPDNet, BTS, and DPdisp.
• The error distributions in Figure 8 (m) and (n) confirm that DDDNet’s inverse depth predictions are consistently closer to ground truth than competing methods.