[Paper Review] Seeing Through the Glass: Neural 3D Reconstruction of Object Inside a Transparent Container

Jinguang Tong, Sundaram Muthu+3 more|arXiv (Cornell University)|Mar 24, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes ReNeuS, a novel neural 3D reconstruction method that explicitly models light reflections and refractions at intersecting glass-air interfaces to recover high-fidelity geometry of objects inside transparent containers. By segmenting the scene into internal and external sub-spaces and using a hybrid ray tracing and volume rendering strategy, ReNeuS achieves state-of-the-art performance on both synthetic and real datasets, outperforming existing methods like NeuS and IDR, especially in reconstructing fine details such as insect wings and antennae.

ABSTRACT

In this paper, we define a new problem of recovering the 3D geometry of an object confined in a transparent enclosure. We also propose a novel method for solving this challenging problem. Transparent enclosures pose challenges of multiple light reflections and refractions at the interface between different propagation media e.g. air or glass. These multiple reflections and refractions cause serious image distortions which invalidate the single viewpoint assumption. Hence the 3D geometry of such objects cannot be reliably reconstructed using existing methods, such as traditional structure from motion or modern neural reconstruction methods. We solve this problem by explicitly modeling the scene as two distinct sub-spaces, inside and outside the transparent enclosure. We use an existing neural reconstruction method (NeuS) that implicitly represents the geometry and appearance of the inner subspace. In order to account for complex light interactions, we develop a hybrid rendering strategy that combines volume rendering with ray tracing. We then recover the underlying geometry and appearance of the model by minimizing the difference between the real and hybrid rendered images. We evaluate our method on both synthetic and real data. Experiment results show that our method outperforms the state-of-the-art (SOTA) methods. Codes and data will be available at https://github.com/hirotong/ReNeuS

Motivation & Objective

• To address the challenge of 3D reconstruction of objects enclosed in transparent containers, which causes severe image distortions due to multiple reflections and refractions.
• To develop a method that overcomes the failure of existing structure-from-motion and neural implicit methods under the violation of the single viewpoint assumption caused by refractive interfaces.
• To enable high-quality 3D reconstruction of fragile museum specimens—such as insects or tissues—preserved in transparent enclosures like resin or glass.
• To explicitly model light interactions across multiple media (air and glass) using a physically-based rendering strategy that accounts for both reflection and refraction.
• To evaluate the method on both synthetic benchmarks and a newly captured real-world dataset of transparently contained specimens.

Proposed method

• The method divides the scene into two sub-spaces: an internal space (inside the transparent container) and an external space (surrounding air), treating them separately for modeling.
• It uses NeuS as the base neural implicit representation to encode geometry and appearance within the internal sub-space.
• A novel hybrid rendering strategy combines volume rendering for the internal object with ray tracing for light paths across the glass-air interfaces to simulate multiple reflections and refractions.
• The system minimizes photometric loss between real images and rendered images to optimize geometry and appearance, with a transmittance-based sparsity prior to suppress false boundaries.
• The transparent container's geometry and pose are assumed known, and background lighting is assumed homogeneous and ambient to simplify the problem.
• The method is trained end-to-end using multi-view images, with the final 3D mesh reconstructed from the learned neural implicit function.

Experimental results

Research questions

• RQ1 Can a neural implicit method accurately reconstruct 3D geometry of an object inside a transparent container when multiple reflections and refractions distort the image?
• RQ2 How does explicitly modeling the internal and external sub-spaces improve reconstruction quality compared to treating the entire scene as a single medium?
• RQ3 To what extent does a hybrid rendering strategy combining volume rendering and ray tracing outperform single-path or single-medium assumptions in handling complex light interactions?
• RQ4 Does modeling multiple reflections and refractions lead to better reconstruction of fine geometric details such as thin wings or antennae?
• RQ5 How robust is the method to violations of the known container geometry and homogeneous lighting assumptions in real-world settings?

Key findings

• On the synthetic dataset, ReNeuS achieves a Chamfer-L1 loss of 16.80, significantly outperforming NeuS (+) at 22.07 and Ours (w/o Trans Loss) at 19.08.
• On the real dataset, ReNeuS successfully reconstructs fine details such as insect wings and antennae, while IDR fails to reconstruct anything beyond a convex hull of the box mask.
• The ablation study shows that the full ReNeuS model achieves zero failure cases in mesh recovery, compared to two failures for the version without transmittance loss and three for NeuS+.
• The hybrid rendering strategy effectively suppresses false boundaries caused by incorrect light path modeling, as seen in the failure of NeuS+ to produce a coherent internal mesh.
• Even on data without a container (i.e., w/o box), ReNeuS outperforms NeuS, suggesting that reflection-induced virtual views improve reconstruction robustness.
• The transmittance-based sparsity prior significantly reduces the number of spurious facets, especially in complex regions like corners and edges.