[Paper Review] Disentangled Latent Transformer for Interpretable Monocular Height Estimation

Zhitong Xiong, Sining Chen+2 more|arXiv (Cornell University)|Jan 17, 2022

Video Surveillance and Tracking Methods6 citations

TL;DR

This paper proposes a disentangled latent Transformer (DLT) network for interpretable monocular height estimation (MHE), leveraging multi-level interpretation—neuron, instance, and attribution analysis—to uncover how deep networks learn semantic and height-selective representations. The DLT model achieves state-of-the-art performance in height estimation and unsupervised semantic segmentation, outperforming existing methods by a large margin while enabling greater model interpretability and reduced computational cost.

ABSTRACT

Monocular height estimation (MHE) from remote sensing imagery has high potential in generating 3D city models efficiently for a quick response to natural disasters. Most existing works pursue higher performance. However, there is little research exploring the interpretability of MHE networks. In this paper, we target at exploring how deep neural networks predict height from a single monocular image. Towards a comprehensive understanding of MHE networks, we propose to interpret them from multiple levels: 1) Neurons: unit-level dissection. Exploring the semantic and height selectivity of the learned internal deep representations; 2) Instances: object-level interpretation. Studying the effects of different semantic classes, scales, and spatial contexts on height estimation; 3) Attribution: pixel-level analysis. Understanding which input pixels are important for the height estimation. Based on the multi-level interpretation, a disentangled latent Transformer network is proposed towards a more compact, reliable, and explainable deep model for monocular height estimation. Furthermore, a novel unsupervised semantic segmentation task based on height estimation is first introduced in this work. Additionally, we also construct a new dataset for joint semantic segmentation and height estimation. Our work provides novel insights for both understanding and designing MHE models.

Motivation & Objective

• To understand how deep neural networks predict height from single monocular remote sensing images by analyzing representations at multiple levels.
• To improve model interpretability in monocular height estimation (MHE), a task critical for disaster response and urban monitoring.
• To design a more compact, reliable, and explainable deep learning model for MHE by leveraging disentangled representations.
• To introduce a novel unsupervised semantic segmentation task using height maps as weak supervision, reducing reliance on costly pixel-level annotations.
• To release a new benchmark dataset for joint semantic segmentation and height estimation to advance the field.

Proposed method

• Proposes a multi-level interpretation framework: neuron-level dissection to study hidden unit selectivity, instance-level analysis to evaluate the impact of semantic class, scale, and spatial context, and pixel-level attribution to identify important input features.
• Designs a disentangled latent Transformer (DLT) architecture that explicitly separates representations for semantic categories and height ranges, enhancing interpretability and model efficiency.
• Introduces a novel unsupervised semantic segmentation head trained using height maps as supervision, enabling weakly supervised learning without manual annotations.
• Employs a self-supervised contrastive learning strategy to learn disentangled representations in the latent space, improving generalization and robustness.
• Utilizes Swin Transformer blocks as the backbone and applies knowledge distillation to compress the model into a smaller, efficient variant with minimal accuracy drop.
• Constructs a new dataset, WDC, combining semantic segmentation and height estimation annotations for training and evaluation.

Experimental results

Research questions

• RQ1 How do individual neurons in MHE networks respond to different semantic objects and height ranges?
• RQ2 What factors—such as object class, scale, or spatial context—most significantly affect height prediction performance?
• RQ3 Can we design a deep learning model that is both accurate and interpretable for monocular height estimation?
• RQ4 To what extent can height maps serve as weak supervision for unsupervised semantic segmentation in remote sensing?
• RQ5 How does the disentangled latent representation improve model performance and interpretability compared to standard Transformers or CNNs?

Key findings

• The DLT model achieves state-of-the-art performance on height estimation, outperforming CNN-based UNet models and matching or exceeding Swin Transformer baselines.
• The proposed DLT model increases the IoU score for unsupervised semantic segmentation from 0.157 (IIC) to 0.306 on the WDC dataset, significantly outperforming MaskContrast, PiCIE, and IIC.
• Hidden units in the MHE network show strong selectivity to both semantic classes (e.g., buildings, trees, roads) and height ranges, indicating disentangled internal representations.
• Semantic class, object scale, and spatial context are the primary factors influencing prediction accuracy, with significant impact on out-of-distribution detection performance.
• The disentangled latent design enables a 16-unit DLT variant that maintains high accuracy while reducing computational complexity, demonstrating model compactness.
• The model enables effective out-of-distribution detection using class-selective neuron activation, improving reliability for risk-sensitive applications.