[Paper Review] Varifocal Multiview Images: Capturing and Visual Tasks

Kejun Wu, Qiong Liu+3 more|arXiv (Cornell University)|Nov 19, 2021

Advanced Vision and Imaging28 references4 citations

TL;DR

This paper introduces varifocal multiview (VFMV) images, a novel imaging model that captures multiple views with distinct focal planes to simultaneously achieve flexible field of view (FoV) and flexible depth of field (DoF). By focusing each view on a different depth plane, VFMV images provide richer focus cues, enhancing 3D scene representation. Experiments show VFMV images detect 2,737 light field features (vs. 1,218 in multiview) and enable higher-quality 3D reconstruction, demonstrating superior performance in visual tasks.

ABSTRACT

Multiview images have flexible field of view (FoV) but inflexible depth of field (DoF). To overcome the limitation of multiview images on visual tasks, in this paper, we present varifocal multiview (VFMV) images with flexible DoF. VFMV images are captured by focusing a scene on distinct depths by varying focal planes, and each view only focused on one single plane.Therefore, VFMV images contain more information in focal dimension than multiview images, and can provide a rich representation for 3D scene by considering both FoV and DoF. The characteristics of VFMV images are useful for visual tasks to achieve high quality scene representation. Two experiments are conducted to validate the advantages of VFMV images in 4D light field feature detection and 3D reconstruction. Experiment results show that VFMV images can detect more light field features and achieve higher reconstruction quality due to informative focus cues. This work demonstrates that VFMV images have definite advantages over multiview images in visual tasks.

Motivation & Objective

• To address the limitation of multiview images, which have flexible FoV but inflexible DoF, by introducing a new imaging model that combines multiview and focal stack advantages.
• To enable high-quality 3D scene representation by incorporating depth-of-field (DoF) information across multiple views through variable focal planes.
• To improve visual task performance in applications such as 3D reconstruction and feature detection by leveraging rich focus cues from distinct focal settings.
• To overcome the trade-off between signal-to-noise ratio (SNR) and DoF in conventional imaging systems by enabling narrow DoF per view with wide FoV across views.

Proposed method

• VFMV images are captured using a multi-camera array where each view is focused on a different focal plane, ensuring high image quality per view due to narrow DoF.
• The system acquires a 9×9 angular arrangement of views, each optimized for a specific depth plane, enabling flexible DoF across the entire scene.
• Light field feature detection is performed using the LiFF algorithm on both VFMV and multiview data, with parameters set to peak threshold 0.0015, non-edge threshold 10, 4 octaves starting from -1, and 3 levels per octave.
• 3D reconstruction is conducted via structure from motion (SfM) using two views per dataset, with SIFT features matched to estimate camera poses and reconstruct 3D point clouds.
• The VFMV acquisition process allows dynamic adjustment of focal planes per view, enabling post-capture refocusing and improved depth representation.
• The method integrates multiview and focal stack principles, creating a hybrid data model that supports both wide FoV and large DoF.

Experimental results

Research questions

• RQ1 Can varifocal multiview images with distinct focal planes per view improve the richness of 4D light field feature detection compared to conventional multiview images?
• RQ2 Does the inclusion of focus cues across multiple focal planes enhance 3D reconstruction quality in visual tasks?
• RQ3 To what extent does the flexible DoF of VFMV images reduce reconstruction artifacts and improve completeness compared to single-focus multiview images?
• RQ4 Can VFMV images effectively resolve the accommodation-vergence conflict in VR/AR by enabling depth-dependent focus per view?

Key findings

• VFMV images detect 2,737 light field features, a significant increase from the 1,218 features detected in conventional multiview images, indicating richer information content.
• The distribution of detected features in VFMV images is more widespread and spatially accurate, attributed to the presence of distinct focus cues across views.
• 3D reconstruction from VFMV images yields a more complete and coherent 3D mesh, with clearer overall structure, whereas multiview reconstruction results are incomplete due to limited DoF.
• The reconstructed scene from VFMV images shows higher quality despite minor flaws, demonstrating the effectiveness of focus cues in improving geometric consistency and feature matching.
• The use of distinct focal planes per view enhances the robustness of feature detection and reconstruction, especially in scenes with complex depth variations.
• The results confirm that VFMV images outperform standard multiview images in visual tasks requiring high-fidelity 3D representation.