[Paper Review] Self-Constructing Graph Convolutional Networks for Semantic Labeling

Qinghui Liu, Michael Kampffmeyer+2 more|arXiv (Cornell University)|Mar 15, 2020

Advanced Neural Network Applications15 references4 citations

TL;DR

This paper proposes Self-Constructing Graph Convolutional Networks (SCG-Net), a novel architecture that learns to self-construct optimized, non-local context graphs directly from input features without prior knowledge. By using learnable latent variables and a variational lower bound with adaptive diagonal enhancement, SCG-Net achieves competitive F1-score (89.8%) on the ISPRS Vaihingen dataset with 14% fewer parameters and lower FLOPs than state-of-the-art models.

ABSTRACT

Graph Neural Networks (GNNs) have received increasing attention in many fields. However, due to the lack of prior graphs, their use for semantic labeling has been limited. Here, we propose a novel architecture called the Self-Constructing Graph (SCG), which makes use of learnable latent variables to generate embeddings and to self-construct the underlying graphs directly from the input features without relying on manually built prior knowledge graphs. SCG can automatically obtain optimized non-local context graphs from complex-shaped objects in aerial imagery. We optimize SCG via an adaptive diagonal enhancement method and a variational lower bound that consists of a customized graph reconstruction term and a Kullback-Leibler divergence regularization term. We demonstrate the effectiveness and flexibility of the proposed SCG on the publicly available ISPRS Vaihingen dataset and our model SCG-Net achieves competitive results in terms of F1-score with much fewer parameters and at a lower computational cost compared to related pure-CNN based work. Our code will be made public soon.

Motivation & Objective

• To address the limitation of existing Graph Neural Networks (GNNs) in semantic labeling due to reliance on manually constructed prior graphs.
• To develop a method that automatically learns optimal graph structures from input features without prior knowledge.
• To integrate self-constructed graphs into deep learning pipelines for improved modeling of long-range dependencies in aerial imagery.
• To achieve competitive performance in semantic labeling with reduced model complexity and computational cost.

Proposed method

• The SCG module learns latent embeddings from 2D feature maps using a variational autoencoder-like framework with reparameterized Gaussian variables.
• A differentiable graph construction process generates a weighted adjacency matrix $\hat{A}$ from latent representations, enabling end-to-end training.
• An adaptive diagonal enhancement method improves neighborhood preservation by adding $\gamma \cdot \text{diag}(A')$ to the adjacency matrix.
• The model uses a variational lower bound loss combining graph reconstruction and Kullback-Leibler divergence regularization.
• A residual prediction head $\hat{\mathbf{y}} = \gamma \cdot \hat{\mathbf{z}}$ is introduced to refine final predictions.
• The SCG-Net integrates the SCG module with a CNN backbone (ResNet50) and a two-layer GCN for final semantic segmentation.

Experimental results

Research questions

• RQ1 Can a graph be self-constructed from raw feature maps without relying on prior knowledge graphs for semantic labeling?
• RQ2 How can a differentiable, end-to-end trainable graph construction mechanism improve long-range context modeling in aerial images?
• RQ3 What is the impact of adaptive diagonal enhancement and variational regularization on graph quality and segmentation performance?
• RQ4 Can a self-constructed graph architecture achieve competitive performance with fewer parameters and lower FLOPs than existing CNN and GCN-based models?

Key findings

• SCG-Net achieves an F1-score of 89.8% on the ISPRS Vaihingen test set, matching the best-performing model DDCM-R50.
• The model uses 8.74 million parameters, which is 12.5% fewer than DDCM-R50’s 9.99 million parameters.
• SCG-Net has a lower computational cost of 4.37 Giga FLOPs compared to DDCM-R50’s 4.86 Giga FLOPs.
• The adaptive diagonal enhancement method improves neighborhood preservation and contributes to better graph structure learning.
• Qualitative results show that SCG-Net predictions are visually close to ground truth, especially in complex object regions.
• The model demonstrates strong generalization and robustness on diverse land cover types, including buildings, trees, and low vegetation.