[Paper Review] Texture Reformer: Towards Fast and Universal Interactive Texture Transfer

Zhizhong Wang, Lei Zhao+5 more|arXiv (Cornell University)|Dec 6, 2021

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Texture Reformer, a fast and universal neural framework for interactive texture transfer using user-specified semantic maps. It employs a feed-forward, multi-view, multi-stage synthesis pipeline with view-specific texture reformation (VSTR) and statistics-based enhancement to achieve high-quality, structure-preserving results up to 2–5 orders of magnitude faster than state-of-the-art methods.

ABSTRACT

In this paper, we present the texture reformer, a fast and universal neural-based framework for interactive texture transfer with user-specified guidance. The challenges lie in three aspects: 1) the diversity of tasks, 2) the simplicity of guidance maps, and 3) the execution efficiency. To address these challenges, our key idea is to use a novel feed-forward multi-view and multi-stage synthesis procedure consisting of I) a global view structure alignment stage, II) a local view texture refinement stage, and III) a holistic effect enhancement stage to synthesize high-quality results with coherent structures and fine texture details in a coarse-to-fine fashion. In addition, we also introduce a novel learning-free view-specific texture reformation (VSTR) operation with a new semantic map guidance strategy to achieve more accurate semantic-guided and structure-preserved texture transfer. The experimental results on a variety of application scenarios demonstrate the effectiveness and superiority of our framework. And compared with the state-of-the-art interactive texture transfer algorithms, it not only achieves higher quality results but, more remarkably, also is 2-5 orders of magnitude faster. Code is available at https://github.com/EndyWon/Texture-Reformer.

Motivation & Objective

• To address the limitations of existing interactive texture transfer methods, which are slow due to iterative optimization and CPU-heavy operations.
• To unify diverse texture transfer tasks—such as doodle-to-art, pattern editing, text effect transfer, and virtual try-on—under a single, universal framework.
• To achieve high-quality results with coherent structures and fine texture details while drastically improving inference speed.
• To develop a learning-free, feed-forward architecture that enables real-time user interaction without retraining for each input.

Proposed method

• The framework uses a coarse-to-fine, three-stage synthesis pipeline: global structure alignment, local texture refinement, and holistic effect enhancement.
• A novel view-specific texture reformation (VSTR) operation is introduced, which uses semantic map guidance to preserve structure and improve alignment without learning.
• VSTR operates in two phases: first a global view captures and aligns source texture structures, then a local view refines fine details.
• Statistics-based enhancement (SE) operations are applied in the final stage to improve low-level effects like color, brightness, and contrast.
• The entire pipeline is implemented using pre-trained auto-encoders and does not require backpropagation or optimization during inference.
• The method is universal and efficient because VSTR and SE are learning-free and the network is feed-forward, enabling real-time performance on CPU or GPU.

Experimental results

Research questions

• RQ1 Can a universal, fast, and high-quality interactive texture transfer framework be built without relying on iterative optimization or learning-based refinement?
• RQ2 How can semantic maps with minimal detail (e.g., doodles) be effectively used to guide accurate texture transfer while preserving local structures?
• RQ3 To what extent can a feed-forward, multi-stage architecture outperform existing optimization-based methods in speed and quality?
• RQ4 Can a learning-free operation like VSTR achieve better structure preservation and semantic alignment than differentiable, learning-based alternatives?

Key findings

• The proposed method achieves 2–5 orders of magnitude faster inference than state-of-the-art methods, with inference times of 0.232 seconds on GPU and 2.573 seconds on CPU for 256×256 images.
• On 512×512 resolution, the method runs in 0.956 seconds on GPU and 12.381 seconds on CPU, significantly outperforming competitors like T-Effect and CFITT.
• User study results show that 32.1% of participants preferred the results of Texture Reformer over other methods, outperforming T-Effect (23.8%), CFITT (26.2%), Neural Doodle (10.3%), and STROTSS (7.6%).
• The method achieves higher perceptual quality, as confirmed by lower LPIPS and style loss, indicating better preservation of source texture structure and holistic style.
• The framework is robust across diverse tasks, including doodle-to-art, pattern editing, text effect transfer, and virtual try-on, demonstrating universal applicability.
• Despite its simplicity, the method shows strong performance, with no need for per-image retraining or complex optimization, making it ideal for interactive applications.