[Paper Review] Unsupervised Enhancement of Real-World Depth Images Using Tri-Cycle GAN

Alona Baruhov, Guy Gilboa|arXiv (Cornell University)|Jan 11, 2020

Advanced Vision and Imaging28 references5 citations

TL;DR

This paper proposes a Tri-Cycle GAN framework to unsupervisedly enhance real-world depth images from low-cost sensors like Intel RealSense R200, using unpaired data from high-quality Kinect 2. By introducing a larger generator, depth-specific masked losses, and a novel Tri-Cycle loss that enforces information preservation in asymmetric domains, the method significantly improves visual and quantitative results over standard Cycle-GAN, especially in handling severe noise, missing pixels, and structured artifacts.

ABSTRACT

Low quality depth poses a considerable challenge to computer vision algorithms. In this work we aim to enhance highly degraded, real-world depth images acquired by a low-cost sensor, for which an analytical noise model is unavailable. In the absence of clean ground-truth, we approach the task as an unsupervised domain-translation between the low-quality sensor domain and a high-quality sensor domain, represented using two unpaired training sets. We employ the highly-successful Cycle-GAN to this task, but find it to perform poorly in this case. Identifying the sources of the failure, we introduce several modifications to the framework, including a larger generator architecture, depth-specific losses that take into account missing pixels, and a novel Tri-Cycle loss which promotes information-preservation while addressing the asymmetry between the domains. We show that the resulting framework dramatically improves over the original Cycle-GAN both visually and quantitatively, extending its applicability to more challenging and asymmetric translation tasks.

Motivation & Objective

• To address the challenge of enhancing highly degraded, real-world depth images from low-cost sensors like Intel RealSense R200, which lack clean ground-truth and analytical noise models.
• To formulate depth enhancement as an unsupervised domain translation task between low-quality and high-quality depth domains without requiring paired training data.
• To overcome the failure of standard Cycle-GAN in this asymmetric, complex degradation setting by introducing architectural and loss function modifications.
• To develop a method that preserves structural details and minimizes artifacts despite severe noise, missing pixels, and non-Gaussian degradation patterns.

Proposed method

• A modified Cycle-GAN framework is used, replacing the standard generator with a larger, deeper architecture to better capture complex depth structures and degradation patterns.
• Depth-specific masked similarity losses are introduced to handle missing pixels and non-uniform data, improving robustness during training.
• A novel Tri-Cycle loss is proposed as a nonlinear generalization of the Moore-Penrose inverse, enabling information preservation in asymmetric domains where cycle consistency fails.
• The framework uses unpaired real-world depth images from RealSense R200 (low-quality) and Kinect 2 (high-quality) for unsupervised training.
• The Tri-Cycle loss enforces that translating a real image to the target domain and back preserves key structural information, even when the domains are not symmetric.
• The method is trained end-to-end using adversarial loss, cycle consistency, and the new Tri-Cycle loss, with no need for paired supervision.

Experimental results

Research questions

• RQ1 Can an unsupervised domain translation framework effectively enhance real-world depth images when no clean ground-truth or noise model is available?
• RQ2 Why does standard Cycle-GAN fail in enhancing depth images with severe, non-Gaussian, and asymmetric degradation patterns?
• RQ3 How can information preservation be enforced in asymmetric domain translation tasks where the inverse mapping is ill-posed or non-existent?
• RQ4 To what extent can architectural scaling and domain-specific losses improve the performance of GAN-based depth enhancement beyond standard Cycle-GAN?
• RQ5 Does the proposed Tri-Cycle loss outperform standard cycle consistency in preserving structural details and reducing artifacts in highly degraded depth images?

Key findings

• The proposed Tri-Cycle GAN significantly outperforms vanilla Cycle-GAN in both visual quality and quantitative metrics, producing sharper edges and fewer artifacts.
• On the DROT dataset, the method achieved a PNCC score of 0.633, outperforming Sparse Depth Sensing (0.611) and vastly surpassing the base Cycle-GAN (0.213).
• The inclusion of the Tri-Cycle loss led to a noticeable reduction in missing pixels and improved geometric accuracy, especially in highly corrupted regions.
• The method successfully preserves fine details and object boundaries, even in regions with strong noise and missing data, due to the depth-specific masked losses.
• The model generalizes well to real-world scenes, producing realistic, detailed depth maps that closely resemble high-quality Kinect 2 outputs.
• Despite its success, the model occasionally reproduces typical Kinect 2 artifacts, such as missing pixels, indicating a trade-off between realism and fidelity.