[Paper Review] 3D Reconstruction from Full-view Fisheye Camera

Chuiwen Ma, Liang Shi+2 more|arXiv (Cornell University)|Jun 20, 2015

Advanced Vision and Imaging3 references20 citations

TL;DR

This paper presents a 3D reconstruction method for full-view fisheye cameras using a spherical camera model and epipolar geometry, enabling sparse and dense reconstruction from multiple Ricoh Theta images. The approach combines manual point correspondence, SIFT matching with RANSAC filtering, and a GUI for interactive visualization, achieving accurate 3D scene reconstruction that preserves real-world structure.

ABSTRACT

In this report, we proposed a 3D reconstruction method for the full-view fisheye camera. The camera we used is Ricoh Theta, which captures spherical images and has a wide field of view (FOV). The conventional stereo apporach based on perspective camera model cannot be directly applied and instead we used a spherical camera model to depict the relation between 3D point and its corresponding observation in the image. We implemented a system that can reconstruct the 3D scene using captures from two or more cameras. A GUI is also created to allow users to control the view perspective and obtain a better intuition of how the scene is rebuilt. Experiments showed that our reconstruction results well preserved the structure of the scene in the real world.

Motivation & Objective

• To enable accurate 3D reconstruction from full-view fisheye cameras, which cannot be handled by standard perspective camera models due to extreme radial distortion.
• To develop a spherical camera model that maps 3D points to a unit sphere and projects them onto a circular image plane.
• To establish a revised epipolar geometry and fundamental matrix estimation for fisheye cameras to support triangulation and camera pose recovery.
• To implement both sparse and dense 3D reconstruction pipelines using point correspondences and disparity mapping.
• To create a GUI for interactive visualization of 360° reconstructed scenes from multiple fisheye images.

Proposed method

• Proposed a spherical camera model where 3D points are projected onto a unit sphere and then vertically projected onto the image plane, using a radial projection model with a fixed focal length.
• Derived the epipolar geometry for fisheye cameras based on the spherical model, enabling the computation of the fundamental matrix from point correspondences.
• Used manually labeled ground truth point pairs to estimate the fundamental matrix, which was then used to filter SIFT feature matches and reduce outliers.
• Applied RANSAC to refine the fundamental matrix and augmented the correspondence pool with high-confidence SIFT matches for denser reconstruction.
• Performed image rectification via equirectangular-to-cube mapping to align corresponding points along the same longitude, enabling traditional stereo disparity computation.
• Generated disparity maps using correlation-based methods on rectified images and used them for dense 3D reconstruction, despite residual distortion.

Experimental results

Research questions

• RQ1 How can a spherical camera model accurately represent the projection of 3D points in full-view fisheye images with 180° field of view?
• RQ2 What modifications to epipolar geometry are required to enable robust fundamental matrix estimation and triangulation in fisheye stereo systems?
• RQ3 Can SIFT feature matching be effectively used for 3D reconstruction in fisheye images, and how can its performance be improved using ground-truth fundamental matrices as filters?
• RQ4 To what extent can dense 3D reconstruction be achieved from fisheye images using rectification and disparity mapping despite residual distortion?
• RQ5 How can a user-friendly GUI be designed to enable interactive exploration of 360° reconstructed scenes from multiple fisheye images?

Key findings

• Sparse 3D reconstruction using manually selected correspondences and the ground-truth fundamental matrix produced accurate results that closely matched the real-world scene structure.
• The integration of SIFT matches filtered by the ground-truth fundamental matrix significantly improved reconstruction density while maintaining geometric consistency.
• Dense reconstruction via disparity mapping was successfully achieved after image rectification, though residual distortion introduced noise into the disparity map.
• The GUI enabled interactive control of viewing perspective across multiple 360° images, enhancing user intuition for the reconstructed 3D scene.
• The method demonstrated robust performance on indoor scenes with structured features like walls and ceilings, but SIFT matching performance degraded in outdoor scenes with repetitive textures or large baseline differences.
• Camera pose estimation (rotation and position) was successfully computed for up to six cameras, enabling multi-view reconstruction with consistent alignment.