[Paper Review] DeepOrganNet: On-the-Fly Reconstruction and Visualization of 3D / 4D Lung Models from Single-View Projections by Deep Deformation Network

Yifan Wang, Zichun Zhong+1 more|arXiv (Cornell University)|Jul 22, 2019

3D Shape Modeling and Analysis50 references4 citations

TL;DR

DeepOrganNet is a deep learning framework that reconstructs high-fidelity, manifold 3D/4D lung meshes from single 2D medical projections (e.g., X-ray or 4D-CT) in real time. It uses a trivariate tensor-product deformation network with a learned latent descriptor to enable on-the-fly, accurate organ modeling, reducing imaging dose and computational time compared to traditional methods.

ABSTRACT

This paper introduces a deep neural network based method, i.e., DeepOrganNet, to generate and visualize high-fidelity 3D / 4D organ geometric models from single-view medical image in real time. Traditional 3D / 4D medical image reconstruction requires near hundreds of projections, which cost insufferable computational time and deliver undesirable high imaging / radiation dose to human subjects. Moreover, it always needs further notorious processes to extract the accurate 3D organ models subsequently. To our knowledge, there is no method directly and explicitly reconstructing multiple 3D organ meshes from a single 2D medical grayscale image on the fly. Given single-view 2D medical images, e.g., 3D / 4D-CT projections or X-ray images, our end-to-end DeepOrganNet framework can efficiently and effectively reconstruct 3D / 4D lung models with a variety of geometric shapes by learning the smooth deformation fields from multiple templates based on a trivariate tensor-product deformation technique, leveraging an informative latent descriptor extracted from input 2D images. The proposed method can guarantee to generate high-quality and high-fidelity manifold meshes for 3D / 4D lung models. The major contributions of this work are to accurately reconstruct the 3D organ shapes from 2D single-view projection, significantly improve the procedure time to allow on-the-fly visualization, and dramatically reduce the imaging dose for human subjects. Experimental results are evaluated and compared with the traditional reconstruction method and the state-of-the-art in deep learning, by using extensive 3D and 4D examples from synthetic phantom and real patient datasets. The proposed method only needs several milliseconds to generate organ meshes with 10K vertices, which has a great potential to be used in real-time image guided radiation therapy (IGRT).

Motivation & Objective

• To address the high radiation dose and long reconstruction times in traditional 3D/4D CBCT imaging, which require hundreds of projections.
• To eliminate the need for post-processing segmentation and reconstruction steps by directly generating high-fidelity 3D organ meshes from a single 2D image.
• To enable real-time, on-the-fly visualization of dynamic lung models during image-guided radiation therapy (IGRT).
• To develop a deep learning framework that reconstructs multiple organ shapes (e.g., left and right lungs) simultaneously from a single-view input.
• To improve shape fidelity and manifold quality compared to existing deep learning methods that produce non-manifold or low-quality meshes.

Proposed method

• The method employs an end-to-end deep neural network with three core components: a feature encoder block, an independent deformation block, and a spatial arrangement (translation) block.
• It uses a trivariate tensor-product deformation technique (free-form deformation) to learn smooth, optimal deformations from multiple anatomical templates.
• A latent descriptor is extracted from the input 2D projection to guide the selection and deformation of the most suitable template.
• The framework jointly learns template selection and deformation fields, enabling accurate reconstruction of complex, patient-specific lung geometries.
• The network is trained on synthetic phantoms and real 4D-CT patient datasets to generalize across diverse anatomical shapes and breathing phases.
• The method supports 4D reconstruction by processing each phase independently, with potential for future RNN and attention-based extensions.

Experimental results

Research questions

• RQ1 Can a deep learning model reconstruct high-fidelity, manifold 3D/4D lung meshes directly from a single 2D medical projection?
• RQ2 Can the proposed method achieve real-time reconstruction (on-the-fly) with sub-second latency for clinical use?
• RQ3 Can the framework simultaneously reconstruct multiple organs (e.g., left and right lungs) from a single input image?
• RQ4 How does the performance of the proposed method compare to traditional FDK and state-of-the-art deep learning methods in terms of shape accuracy and mesh quality?
• RQ5 Can the method significantly reduce imaging dose by replacing multi-projection CBCT with single-view imaging while maintaining diagnostic accuracy?

Key findings

• DeepOrganNet reconstructs 3D/4D lung meshes with approximately 10,000 vertices in just a few milliseconds, achieving real-time performance.
• The method reduces reconstruction time from minutes (traditional FDK) to under 22 milliseconds per 4D phase, enabling on-the-fly visualization in IGRT.
• The reconstructed meshes are manifold and high-fidelity, outperforming existing deep learning methods that produce non-manifold or invalid surface elements.
• The framework successfully captures dynamic breathing motion, with deformation magnitudes accurately mapped across expiration phases in both phantom and real patient data.
• The method achieves superior shape reconstruction accuracy compared to both traditional FDK and state-of-the-art deep learning approaches, especially in low-projection scenarios.
• The system demonstrates feasibility for clinical deployment, with an official clinical trial currently being arranged with a collaborating hospital.