[Paper Review] CutDepth:Edge-aware Data Augmentation in Depth Estimation

Yasunori Ishii, Takayoshi Yamashita|arXiv (Cornell University)|Jul 16, 2021

Advanced Vision and Imaging14 references18 citations

TL;DR

CutDepth proposes an edge-aware data augmentation method for monocular depth estimation that pastes cropped depth maps onto RGB images during training, preserving structural consistency while enhancing feature diversity. The method improves depth estimation accuracy—especially at long distances—by regularizing the latent space, outperforming conventional augmentations like CutMix and CutOut in both objective and subjective evaluations.

ABSTRACT

It is difficult to collect data on a large scale in a monocular depth estimation because the task requires the simultaneous acquisition of RGB images and depths. Data augmentation is thus important to this task. However, there has been little research on data augmentation for tasks such as monocular depth estimation, where the transformation is performed pixel by pixel. In this paper, we propose a data augmentation method, called CutDepth. In CutDepth, part of the depth is pasted onto an input image during training. The method extends variations data without destroying edge features. Experiments objectively and subjectively show that the proposed method outperforms conventional methods of data augmentation. The estimation accuracy is improved with CutDepth even though there are few training data at long distances.

Motivation & Objective

• To address the challenge of limited real-world data in monocular depth estimation by introducing a novel data augmentation strategy.
• To improve depth estimation performance, especially for distant objects, using minimal training data.
• To maintain structural consistency in edge features during augmentation, avoiding excessive geometric distortion.
• To regularize the latent space by aligning RGB and depth representations through augmented inputs.
• To evaluate the effectiveness of the method in terms of diversity, affinity, and estimation accuracy.

Proposed method

• CutDepth performs data augmentation by replacing a random region of the input RGB image with a corresponding cropped region from the ground-truth depth map.
• The replacement region is determined by random parameters a, b, c, d drawn from a uniform distribution, with size controlled by a hyperparameter p.
• The blending is performed via a binary mask M, where x′_s = M * x_s + (1 - M) * x_t, ensuring spatial alignment between RGB and depth features.
• When input and depth channels differ (C_s ≠ C_t), they are concatenated along the channel dimension to maintain compatibility.
• The method preserves edge features by ensuring spatial correspondence between depth and RGB edges, minimizing low-level feature disruption.
• The approach is evaluated using standard depth estimation metrics and latent space distance measures (RMSE, MAE, cosine similarity).

Experimental results

Research questions

• RQ1 How does CutDepth improve depth estimation accuracy compared to conventional data augmentation methods?
• RQ2 To what extent does CutDepth preserve edge features and structural consistency during augmentation?
• RQ3 How does CutDepth affect the diversity and affinity of the data distribution in the latent space?
• RQ4 Can CutDepth effectively regularize the latent representation between RGB and depth features?
• RQ5 Does CutDepth maintain performance under low-data regimes, especially for long-distance depth estimation?

Key findings

• CutDepth outperformed baseline, CutOut, RE, and CutMix across all metrics on the KITTI dataset, with the best performance in Abs Rel (0.136), log10 (0.055), RMSE (0.278), and d1 (92.8%).
• At p = 0.75, CutDepth maintained stable performance, while conventional methods like CutOut and RE degraded significantly with increasing p.
• With only 25% of the training data, CutDepth achieved a d1 score of 89.7%, surpassing the baseline (87.2%) and CutMix (88.9%).
• The cosine distance between RGB and depth features in the latent space was highest for CutDepth (0.37 at p=0.25), indicating stronger feature alignment and regularization.
• CutDepth demonstrated superior diversity and affinity in data distribution, with a broader spread and smaller deviation from the original data than baseline and conventional methods.
• Subjective evaluation confirmed that CutDepth produced sharper object contours and better long-distance depth estimation than baseline and CutMix.