[Paper Review] Learning Graph Neural Networks for Image Style Transfer

Yongcheng Jing, Yining Mao+5 more|arXiv (Cornell University)|Jul 24, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a semi-parametric neural style transfer framework using graph neural networks (GNNs) to establish learnable, fine-grained content-style correspondences at the local patch level. By modeling style transfer as attention-based message passing between content and style nodes in a heterogeneous graph, the method reduces artifacts from patch mismatching and preserves both global style coherence and local details, achieving superior stylization with flexible, diversified output via dynamic edge control at inference.

ABSTRACT

State-of-the-art parametric and non-parametric style transfer approaches are prone to either distorted local style patterns due to global statistics alignment, or unpleasing artifacts resulting from patch mismatching. In this paper, we study a novel semi-parametric neural style transfer framework that alleviates the deficiency of both parametric and non-parametric stylization. The core idea of our approach is to establish accurate and fine-grained content-style correspondences using graph neural networks (GNNs). To this end, we develop an elaborated GNN model with content and style local patches as the graph vertices. The style transfer procedure is then modeled as the attention-based heterogeneous message passing between the style and content nodes in a learnable manner, leading to adaptive many-to-one style-content correlations at the local patch level. In addition, an elaborated deformable graph convolutional operation is introduced for cross-scale style-content matching. Experimental results demonstrate that the proposed semi-parametric image stylization approach yields encouraging results on the challenging style patterns, preserving both global appearance and exquisite details. Furthermore, by controlling the number of edges at the inference stage, the proposed method also triggers novel functionalities like diversified patch-based stylization with a single model.

Motivation & Objective

• To address the limitations of parametric methods, which fail to preserve fine-grained local style patterns due to global statistics alignment.
• To overcome the artifacts in non-parametric methods caused by one-to-one patch matching and content-style mismatching.
• To develop a semi-parametric framework that enables adaptive, many-to-one style-content correspondence learning at the patch level.
• To achieve flexible, diversified stylization outputs using a single trained model by controlling edge connections during inference.
• To enhance cross-scale style-content matching through a deformable graph convolutional operation.

Proposed method

• Constructs a heterogeneous graph where content and style feature patches are treated as distinct node types.
• Employs graph attention networks (GAT) for attention-based message passing, aggregating the k most similar style patches for each content node.
• Introduces a deformable graph convolutional layer with a learnable scale predictor to enable cross-scale feature matching and spatially adaptive stroke generation.
• Incorporates intra-domain edges between content nodes to maintain global stylization coherence across semantically similar regions.
• Uses normalized cross-correlation (NCC) as the similarity metric for building heterogeneous edges, outperforming Euclidean distance.
• Enables diversified stylization by dynamically adjusting the number of connections per node during inference, allowing multiple unique outputs from a single model.

Experimental results

Research questions

• RQ1 Can a learnable, many-to-one correspondence between content and style patches reduce artifacts caused by patch mismatching in non-parametric style transfer?
• RQ2 How can GNNs effectively model fine-grained, adaptive content-style relationships for improved local detail preservation?
• RQ3 Can deformable graph convolutions enable cross-scale style-content matching for more natural and spatially adaptive stroke generation?
• RQ4 To what extent can a single GNN-based model generate diverse stylization outputs by controlling inference-time connectivity?
• RQ5 Does using NCC instead of Euclidean distance as the similarity metric improve the quality of content-style correspondence in the graph?

Key findings

• The proposed GNN-based method achieves superior stylization performance on both global stroke arrangement and local fine-grained patterns, outperforming state-of-the-art parametric and non-parametric methods.
• Using normalized cross-correlation (NCC) as the similarity metric yields significantly better results than Euclidean distance, particularly in preserving local details and global coherence.
• The deformable graph convolutional module enables spatially adaptive, multi-scale stroke generation, enhancing contrast and semantic saliency in stylized outputs.
• Incorporating intra-domain edges between content nodes improves global consistency, especially in semantically similar regions such as eyes and backgrounds.
• The model supports diversified, patch-based stylization with a single inference pass by varying the number of connections per node, enabling user-controlled style variation.
• Larger patch sizes lead to larger, more prominent strokes in the stylized output, demonstrating controllability over stroke scale.