[Paper Review] LayoutGMN: Neural Graph Matching for Structural Layout Similarity

Akshay Gadi Patil, Manyi Li+3 more|arXiv (Cornell University)|Dec 11, 2020

Advanced Image and Video Retrieval Techniques46 references4 citations

TL;DR

LayoutGMN is a deep neural network that learns structural layout similarity via attention-based graph matching in a triplet network setting, using IoU-based weak supervision to train a model with built-in structural bias. It outperforms IoU and state-of-the-art baselines in retrieval accuracy, better aligning with human judgment of structural similarity in floorplans and UI designs.

ABSTRACT

We present a deep neural network to predict structural similarity between 2D layouts by leveraging Graph Matching Networks (GMN). Our network, coined LayoutGMN, learns the layout metric via neural graph matching, using an attention-based GMN designed under a triplet network setting. To train our network, we utilize weak labels obtained by pixel-wise Intersection-over-Union (IoUs) to define the triplet loss. Importantly, LayoutGMN is built with a structural bias which can effectively compensate for the lack of structure awareness in IoUs. We demonstrate this on two prominent forms of layouts, viz., floorplans and UI designs, via retrieval experiments on large-scale datasets. In particular, retrieval results by our network better match human judgement of structural layout similarity compared to both IoUs and other baselines including a state-of-the-art method based on graph neural networks and image convolution. In addition, LayoutGMN is the first deep model to offer both metric learning of structural layout similarity and structural matching between layout elements.

Motivation & Objective

• To develop a deep learning model that measures structural similarity between 2D layouts such as floorplans and UI designs, beyond pixel-level metrics.
• To address the limitation of IoU in capturing structural correspondence by embedding structural inductive bias into the model.
• To enable both metric learning for layout similarity and element-level structural matching via attention mechanisms.
• To train the model with weak supervision using IoU thresholds, avoiding costly human annotations.
• To demonstrate superior retrieval performance that aligns with human judgment of structural similarity.

Proposed method

• Represents each layout as a directed, fully connected graph with nodes encoding element semantics, positions, and edges for structural relationships.
• Employs an attention-based Graph Matching Network (GMN) for cross-graph message passing and node-level matching within a triplet loss framework.
• Uses triplet loss with weak labels derived from IoU thresholds to train the network, enabling end-to-end learning without expert annotations.
• Introduces structural inductive bias via attention mechanisms that enforce correspondence between layout elements during graph embedding learning.
• Leverages learned attention weights for both similarity prediction and element-level matching, enabling label transfer across layouts.
• Trains the model in a weakly supervised manner, using IoU as a proxy for structural similarity, while the model's architecture compensates for IoU's lack of structure awareness.

Experimental results

Research questions

• RQ1 Can a deep neural network learn a layout similarity metric that better reflects human perception of structural similarity than pixel-based metrics like IoU?
• RQ2 To what extent can a graph matching network with attention mechanisms learn structural correspondence between layout elements without explicit structural supervision?
• RQ3 How effective is weak supervision via IoU thresholds in training a model that generalizes to structural layout similarity beyond pixel alignment?
• RQ4 Can the attention mechanism in LayoutGMN be used to transfer labels from one layout to another, even under geometric misalignment?
• RQ5 Does LayoutGMN outperform state-of-the-art GNN-based and image-based models in layout retrieval tasks for floorplans and UI designs?

Key findings

• LayoutGMN achieves higher Precision@k scores than IoU and other baselines on large-scale floorplan and UI design datasets, with Precision@10 of 11.3% when edges are preserved.
• The model's retrieval results are more consistent with human judgment of structural similarity than IoU, particularly in cases where IoU fails due to misaligned but structurally similar layouts.
• Ablation studies show that removing graph edges, box positions, or node semantics significantly degrades performance, with Precision@1 dropping to 15% when position encoding is removed.
• LayoutGMN successfully transfers labels from a source to a target layout using attention-based element matching, even when layouts are not geometrically aligned.
• The attention mechanism enables robust element-level correspondence, as demonstrated in label transfer where LayoutGMN outperforms maximum pixel-overlap matching.
• The model’s structural bias effectively compensates for the lack of structure awareness in IoU-based supervision, allowing it to learn meaningful structural similarity despite weak labels.