[Paper Review] Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization

Zhe Chen, Wenhai Wang+3 more|arXiv (Cornell University)|Mar 22, 2021

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes URST, a novel framework for ultra-high-resolution neural style transfer that processes arbitrary-resolution images (e.g., 10000×10000) by dividing them into patches and applying a new Thumbnail Instance Normalization (TIN) layer to maintain style consistency across patches. It achieves state-of-the-art results with 15× lower memory usage than prior methods and enables large brush strokes via a stroke perceptual loss, all without retraining existing models.

ABSTRACT

We present an extremely simple Ultra-Resolution Style Transfer framework, termed URST, to flexibly process arbitrary high-resolution images (e.g., 10000x10000 pixels) style transfer for the first time. Most of the existing state-of-the-art methods would fall short due to massive memory cost and small stroke size when processing ultra-high resolution images. URST completely avoids the memory problem caused by ultra-high resolution images by (1) dividing the image into small patches and (2) performing patch-wise style transfer with a novel Thumbnail Instance Normalization (TIN). Specifically, TIN can extract thumbnail features' normalization statistics and apply them to small patches, ensuring the style consistency among different patches. Overall, the URST framework has three merits compared to prior arts. (1) We divide input image into small patches and adopt TIN, successfully transferring image style with arbitrary high-resolution. (2) Experiments show that our URST surpasses existing SOTA methods on ultra-high resolution images benefiting from the effectiveness of the proposed stroke perceptual loss in enlarging the stroke size. (3) Our URST can be easily plugged into most existing style transfer methods and directly improve their performance even without training. Code is available at https://git.io/URST.

Motivation & Objective

• To address the memory limitations and small stroke size issues in existing neural style transfer methods when processing ultra-high-resolution images.
• To enable arbitrary-resolution style transfer on standard GPUs by avoiding full-image processing.
• To maintain visual consistency across patches during stylization, which is often compromised in patch-wise approaches.
• To improve stylization quality on ultra-high-resolution outputs by promoting larger, more perceptually pleasing brush strokes.

Proposed method

• Divides ultra-high-resolution content images into small, non-overlapping patches to reduce GPU memory consumption.
• Introduces Thumbnail Instance Normalization (TIN), which computes normalization statistics (mean and variance) from a low-resolution thumbnail of each patch to guide style transfer.
• Applies TIN to normalize features in each patch using statistics derived from a thumbnail, ensuring consistent style across the entire image.
• Incorporates a stroke perceptual loss ($\mathcal{L}_{\rm sp}$) as an auxiliary loss to encourage the network to learn larger, sparser brush strokes.
• Uses a patch-wise stylization pipeline: divide → stylize with TIN and $\mathcal{L}_{\rm sp}$ → assemble patches into a full-resolution output.
• Supports plug-and-play integration with most existing instance normalization (IN) or instance-wise normalization (IW)-based style transfer models without retraining.

Experimental results

Research questions

• RQ1 Can ultra-high-resolution style transfer be achieved on standard GPUs without exceeding memory limits?
• RQ2 How can style consistency be preserved across patches when processing large images in a patch-wise manner?
• RQ3 Can the stroke size in stylized outputs be effectively enlarged to reduce dense texture artifacts in high-resolution results?
• RQ4 Does the proposed TIN layer provide a stable and accurate approximation of full-image instance normalization for high-resolution inputs?
• RQ5 Can the stroke perceptual loss improve stylization quality without modifying the underlying model architecture?

Key findings

• URST achieves 15× memory reduction compared to the prior state-of-the-art method (Wang et al., 2020), requiring only 1.94GB GPU memory for a 10000×10000 image, while the baseline requires 30.25GB.
• The framework successfully processes a 12000×8000 pixel image (96 megapixels) using just 2.5GB of GPU memory on a single 12GB GPU.
• TIN with a thumbnail size of at least 1024×1024 pixels produces normalization statistics that are very close to full-image instance normalization, ensuring high-quality stylization.
• The stroke perceptual loss significantly increases brush stroke size, resulting in more natural and visually pleasing outputs compared to baseline methods.
• URST improves performance across multiple existing style transfer models without retraining, demonstrating strong generalization and plug-in compatibility.
• The method produces high-quality stylized results with large, sparse brush strokes, avoiding the dense texture artifacts common in small-stroke approaches.