[Paper Review] Edge-Semantic Learning Strategy for Layout Estimation in Indoor Environment

Weidong Zhang, Wei Zhang+1 more|arXiv (Cornell University)|Jan 3, 2019

Advanced Image and Video Retrieval Techniques34 references4 citations

TL;DR

This paper proposes a novel edge-semantic learning strategy for monocular indoor layout estimation using a shared-encoder, dual-decoder network that jointly predicts edge maps and semantic labels. By combining these predictions in a scoring function and refining layout hypotheses via ray sampling and a predefined layout pool, the method achieves state-of-the-art performance on benchmark datasets, outperforming prior approaches by up to 2.73% in pixel error and 1.13% in corner error.

ABSTRACT

Visual cognition of the indoor environment can benefit from the spatial layout estimation, which is to represent an indoor scene with a 2D box on a monocular image. In this paper, we propose to fully exploit the edge and semantic information of a room image for layout estimation. More specifically, we present an encoder-decoder network with shared encoder and two separate decoders, which are composed of multiple deconvolution (transposed convolution) layers, to jointly learn the edge maps and semantic labels of a room image. We combine these two network predictions in a scoring function to evaluate the quality of the layouts, which are generated by ray sampling and from a predefined layout pool. Guided by the scoring function, we apply a novel refinement strategy to further optimize the layout hypotheses. Experimental results show that the proposed network can yield accurate estimates of edge maps and semantic labels. By fully utilizing the two different types of labels, the proposed method achieves state-of-the-art layout estimation performance on benchmark datasets.

Motivation & Objective

• To improve indoor layout estimation accuracy by jointly leveraging edge and semantic information from monocular images.
• To address the limitations of existing methods that rely on either edge maps or semantic labels in isolation, which often suffer from low-quality predictions and poor robustness.
• To develop a robust layout proposal generation strategy that reduces dependency on vanishing point detection, which is sensitive to noise and occlusion.
• To introduce a pixel-level optimization strategy that refines layout hypotheses using combined edge and semantic predictions.
• To demonstrate that edge and semantic predictions can mutually compensate for each other, enhancing overall layout estimation robustness.

Proposed method

• A dual-decoder encoder-decoder network is designed with a shared encoder (based on VGG-16) and two separate decoders using deconvolutional layers to predict edge maps and semantic segmentation maps in parallel.
• The edge map represents boundaries of walls, floor, and ceiling, while the semantic labels represent five surfaces: ceiling, floor, front wall, left wall, and right wall.
• A scoring function combines the predicted edge and semantic maps to evaluate and rank layout hypotheses generated via ray sampling and from a predefined layout pool.
• Layout hypotheses are refined through an optimization step guided by the scoring function, improving accuracy and robustness.
• The method uses a predefined layout pool to enhance robustness, especially in cluttered or occluded scenes where vanishing point detection fails.
• Joint training of edge and semantic heads enables mutual improvement, with each modality compensating for the other’s weaknesses during inference.

Experimental results

Research questions

• RQ1 Can joint learning of edge maps and semantic labels improve layout estimation accuracy compared to learning them independently?
• RQ2 How does the combination of edge and semantic predictions enhance robustness in the presence of occlusions and clutter?
• RQ3 Can a predefined layout pool combined with ray sampling generate more reliable layout hypotheses than vanishing point-based methods?
• RQ4 To what extent does the proposed optimization strategy improve final layout prediction beyond initial hypotheses?
• RQ5 Does the mutual compensation between edge and semantic predictions lead to better generalization across diverse indoor scene layouts?

Key findings

• The proposed method achieves state-of-the-art performance on the LSUN and Hedau benchmark datasets, outperforming previous methods by 2.73% in pixel error and 1.13% in corner error on LSUN.
• Joint training of edge and semantic heads reduces both edge error and semantic error compared to single-task training, demonstrating mutual benefit during learning.
• The method produces highly accurate edge and semantic predictions even in challenging scenes with severe occlusions, such as when walls are hidden by furniture or doors.
• The use of a predefined layout pool significantly improves robustness, especially in scenes with non-standard viewpoints or complex structures.
• Adding the layout pool hypotheses (Lp) to ray-sampled hypotheses (Lv) improves both layout error and layout type accuracy, confirming the value of prior knowledge on layout similarity.
• The proposed optimization strategy leads to measurable accuracy gains, showing that refinement based on combined edge and semantic scores enhances final layout quality.