[Paper Review] EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images

Changha Shin, Hae‐Gon Jeon+3 more|arXiv (Cornell University)|Apr 6, 2018

Advanced Vision and Imaging7 references17 citations

TL;DR

EPINET proposes a fully-convolutional neural network that leverages epipolar geometry to enable fast and accurate depth estimation from light field images. By using a multi-stream architecture to process four angular directions and applying light-field-specific data augmentation, the method achieves state-of-the-art performance on the HCI 4D Light Field Benchmark, ranking first in accuracy and speed—85× faster than the second-best method with superior depth map quality.

ABSTRACT

Light field cameras capture both the spatial and the angular properties of light rays in space. Due to its property, one can compute the depth from light fields in uncontrolled lighting environments, which is a big advantage over active sensing devices. Depth computed from light fields can be used for many applications including 3D modelling and refocusing. However, light field images from hand-held cameras have very narrow baselines with noise, making the depth estimation difficult. any approaches have been proposed to overcome these limitations for the light field depth estimation, but there is a clear trade-off between the accuracy and the speed in these methods. In this paper, we introduce a fast and accurate light field depth estimation method based on a fully-convolutional neural network. Our network is designed by considering the light field geometry and we also overcome the lack of training data by proposing light field specific data augmentation methods. We achieved the top rank in the HCI 4D Light Field Benchmark on most metrics, and we also demonstrate the effectiveness of the proposed method on real-world light-field images.

Motivation & Objective

• To address the challenge of accurate and efficient depth estimation from hand-held light field cameras with narrow baselines and noisy sub-aperture images.
• To overcome the data scarcity problem in training deep learning models for light field depth estimation by introducing light-field-specific data augmentation techniques.
• To design a fully-convolutional neural network that explicitly incorporates epipolar geometry to improve depth map quality and spatial consistency.
• To achieve both high accuracy and real-time inference speed, balancing the trade-off between performance and computational efficiency in light field depth estimation.

Proposed method

• The network employs four identical, parallel processing streams to extract features from four angular directions (horizontal, vertical, and two diagonals) of sub-aperture images, enabling directional invariance and improved geometric reasoning.
• Features from the four streams are fused through a merging network to produce a high-level representation suitable for depth prediction.
• The method uses a fully-convolutional architecture to enable end-to-end, efficient inference with sub-pixel accuracy on full-resolution light field inputs.
• A novel data augmentation strategy is introduced, including view-shifting, rotation, scaling, transpose, and color jittering, specifically tailored for light field image properties to increase training data diversity and model robustness.
• The network is trained end-to-end using supervised learning on light field datasets, with loss functions optimized for disparity accuracy and edge preservation.
• Post-processing includes a weighted median filter to remove sparse disparity errors in real-world data, improving final depth map quality.

Experimental results

Research questions

• RQ1 Can a fully-convolutional neural network effectively exploit epipolar geometry to improve depth estimation accuracy in light field images?
• RQ2 How can data scarcity in light field depth estimation be mitigated using domain-specific data augmentation strategies?
• RQ3 To what extent can a multi-stream architecture that processes angular directions independently improve depth map quality and boundary sharpness?
• RQ4 Does the proposed method achieve both high accuracy and real-time inference speed, outperforming existing optimization-based and deep learning-based approaches?

Key findings

• EPINET achieved the top rank on the HCI 4D Light Field Benchmark across most metrics, including bad pixel ratio, mean square error, and runtime.
• The method is 85 times faster than OBER-cross+ANP, the second-best method in accuracy, while maintaining superior depth quality.
• The multi-stream architecture enables sharp object boundary detection, such as on airplane wings and toy heads, due to directional matching consistency.
• The proposed data augmentation significantly improves model generalization and performance, especially in noisy scenes like the Dots scene with spatially varying noise.
• On real-world light-field images from the Lytro Illum camera, EPINET with 9×9 angular resolution produced metric-scale 3D reconstructions with high fidelity, outperforming prior methods in speed and accuracy.
• The network maintains strong performance on textureless and reflective regions despite limitations, suggesting room for future integration of photometric or material priors.