[Paper Review] Photo Stylistic Brush: Robust Style Transfer via Superpixel-Based Bipartite Graph

Jiaying Liu, Wenhan Yang+2 more|arXiv (Cornell University)|Jun 13, 2016

Generative Adversarial Networks and Image Synthesis8 references3 citations

TL;DR

This paper proposes SuperBIG, a robust exemplar-based photo stylization method that uses a two-step superpixel-based bipartite graph approach to transfer complex visual styles—such as color, tone, and contrast—between images. By leveraging hierarchical features and global superpixel-level matching, SuperBIG achieves superior visual quality and consistency, outperforming state-of-the-art methods in both qualitative and quantitative evaluations, especially on challenging cases like night scenes.

ABSTRACT

With the rapid development of social network and multimedia technology, customized image and video stylization has been widely used for various social-media applications. In this paper, we explore the problem of exemplar-based photo style transfer, which provides a flexible and convenient way to invoke fantastic visual impression. Rather than investigating some fixed artistic patterns to represent certain styles as was done in some previous works, our work emphasizes styles related to a series of visual effects in the photograph, e.g. color, tone, and contrast. We propose a photo stylistic brush, an automatic robust style transfer approach based on Superpixel-based BIpartite Graph (SuperBIG). A two-step bipartite graph algorithm with different granularity levels is employed to aggregate pixels into superpixels and find their correspondences. In the first step, with the extracted hierarchical features, a bipartite graph is constructed to describe the content similarity for pixel partition to produce superpixels. In the second step, superpixels in the input/reference image are rematched to form a new superpixel-based bipartite graph, and superpixel-level correspondences are generated by a bipartite matching. Finally, the refined correspondence guides SuperBIG to perform the transformation in a decorrelated color space. Extensive experimental results demonstrate the effectiveness and robustness of the proposed method for transferring various styles of exemplar images, even for some challenging cases, such as night images.

Motivation & Objective

• To address the limitations of existing style transfer methods that focus on isolated elements like color or texture, often failing to preserve holistic visual effects in photographs.
• To develop a flexible, automatic tool for non-expert users to transfer rich, multi-faceted photographic styles from a reference image to an input image.
• To overcome the separability assumption between content and style by modeling style as a holistic, content-aware visual effect involving color, contrast, and lighting.
• To improve robustness and visual quality in style transfer, especially for challenging inputs such as low-light or high-contrast photographs.

Proposed method

• Construct a pixel-level bipartite graph using dense correspondence and hierarchical features (color, distance, texture, patch intensity, gradient) to partition input and reference images into superpixels.
• Apply a two-step bipartite graph algorithm: first, use hierarchical features to form initial superpixels via content similarity; second, perform superpixel-level bipartite matching to establish global correspondences.
• Refine the correspondence by re-matching superpixels globally, ensuring consistency across the entire image and reducing local artifacts.
• Perform style transformation in a decorrelated color space guided by the refined superpixel-level correspondences to decouple color and luminance effects.
• Use confidence scores from deep matching to prioritize reliable correspondences and improve robustness in ambiguous regions.
• Integrate multiple primitive features (color, distance, texture, patch intensity, gradient) in a hierarchical feature composition to balance local consistency and global coherence.

Experimental results

Research questions

• RQ1 Can a superpixel-based bipartite graph approach improve the robustness and visual quality of exemplar-based photo style transfer compared to pixel-level or patch-based methods?
• RQ2 How effective is the two-step bipartite graph framework in preserving holistic photographic styles, including color, tone, and contrast, across diverse image conditions?
• RQ3 To what extent do hierarchical features (color, texture, gradient, etc.) contribute to accurate superpixel formation and correspondence matching?
• RQ4 Can the method generalize to challenging photographic inputs such as night scenes or images with complex lighting?

Key findings

• SuperBIG achieved the lowest average ranking score (1.15) in the user study, significantly outperforming six competing methods in perceptual quality and style consistency.
• The ablation study confirmed that combining multiple hierarchical features—especially gradient and patch intensity—significantly improved visual quality and reduced false transfers.
• The superpixel-level bipartite matching step reduced local inconsistencies and artifacts, such as textured skies and incorrect color regions, that were present in pixel-level correspondence results.
• The method demonstrated strong robustness on challenging night images, where other methods failed to preserve style coherence or produced unnatural results.
• The use of hierarchical features and global optimization led to more accurate and visually plausible style transfer, as validated by both quantitative metrics and user preference studies.