[Paper Review] Learning deep structured active contours end-to-end

Diego Marcos, Devis Tuia+5 more|arXiv (Cornell University)|Mar 16, 2018

Automated Road and Building ExtractionEngineering29 references43 citations

TL;DR

DSAC integrates local geometric priors into an end-to-end trainable framework by predicting energy terms for an active contour model with a CNN, achieving improved building instance segmentation over state-of-the-art on multiple datasets.

ABSTRACT

The world is covered with millions of buildings, and precisely knowing each instance's position and extents is vital to a multitude of applications. Recently, automated building footprint segmentation models have shown superior detection accuracy thanks to the usage of Convolutional Neural Networks (CNN). However, even the latest evolutions struggle to precisely delineating borders, which often leads to geometric distortions and inadvertent fusion of adjacent building instances. We propose to overcome this issue by exploiting the distinct geometric properties of buildings. To this end, we present Deep Structured Active Contours (DSAC), a novel framework that integrates priors and constraints into the segmentation process, such as continuous boundaries, smooth edges, and sharp corners. To do so, DSAC employs Active Contour Models (ACM), a family of constraint- and prior-based polygonal models. We learn ACM parameterizations per instance using a CNN, and show how to incorporate all components in a structured output model, making DSAC trainable end-to-end. We evaluate DSAC on three challenging building instance segmentation datasets, where it compares favorably against state-of-the-art. Code will be made available.

Motivation & Objective

• Bridge CNN-based feature learning with geometric shape priors for precise building footprints.
• Enable end-to-end training of an active contour model by learning its energy terms.
• Incorporate local, image-dependent priors including balloon, curvature, and data terms into a unified framework.
• Evaluate DSAC on multiple building datasets and compare to state-of-the-art.

Proposed method

• Represent buildings as polygons and learn a per-instance energy function for an ACM via a CNN.
• Predict D (data term), alpha (length penalty), beta (curvature penalty), and kappa (balloon) as image-dependent maps.
• Incorporate local priors into the ACM energy and optimize via a structured SVM loss in an end-to-end training loop.
• Infer contours with a differentiable-like update y^{t+1} = (I + A + B)^{-1}(y^{t} - dE_ext/dy^{t}).
• Train with loss-augmented inference using IoU-based task loss Δ and a max-margin SSVM objective, updating CNN weights by backpropagation.
• Utilize a locally penalized ACM where D, alpha, beta, kappa are learned maps (per-pixel) rather than global constants.

Experimental results

Research questions

• RQ1 Can local, image-dependent priors improve the geometric accuracy of instance segmentation for buildings?
• RQ2 Does integrating an ACM with a CNN in an end-to-end trainable framework yield better instance footprints than purely CNN-based baselines?
• RQ3 What is the impact of including the balloon term and per-pixel curvature/length penalties on segmentation quality?
• RQ4 How does end-to-end SSVM-based training with loss-augmented inference affect learning of energy terms?

Key findings

• DSAC achieves higher IoU than the CNN baseline on Vaihingen and Bing huts in manual initialization scenarios (IoU improvements reported in Table 1).
• Local per-pixel beta and balloon priors improve segmentation quality, while a global kappa or omitting kappa degrades performance.
• In TorontoCity automatic initialization, DSAC improves weighted coverage and shape similarity over the referenced methods (DWT, FCN, ResNet).
• Hybrid initialization (training with raw DWT and testing with post-processed DWT) yields the best WeighCov/PolySim in TorontoCity experiments.
• DSAC demonstrates the importance of the balloon term for contour convergence and of local priors for sharp corners and straight edges.