[Paper Review] Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer

Xide Xia, Meng Zhang+5 more|arXiv (Cornell University)|Apr 23, 2020

Generative Adversarial Networks and Image Synthesis26 references5 citations

TL;DR

This paper proposes a real-time, photorealistic style transfer method using a feed-forward neural network that learns local edge-aware affine transforms in bilateral space, guaranteeing photorealism. The approach achieves 4K real-time performance on mobile devices—three orders of magnitude faster than prior methods—while producing sharp, artifact-free results that degrade gracefully on unseen inputs.

ABSTRACT

Photorealistic style transfer is the task of transferring the artistic style of an image onto a content target, producing a result that is plausibly taken with a camera. Recent approaches, based on deep neural networks, produce impressive results but are either too slow to run at practical resolutions, or still contain objectionable artifacts. We propose a new end-to-end model for photorealistic style transfer that is both fast and inherently generates photorealistic results. The core of our approach is a feed-forward neural network that learns local edge-aware affine transforms that automatically obey the photorealism constraint. When trained on a diverse set of images and a variety of styles, our model can robustly apply style transfer to an arbitrary pair of input images. Compared to the state of the art, our method produces visually superior results and is three orders of magnitude faster, enabling real-time performance at 4K on a mobile phone. We validate our method with ablation and user studies.

Motivation & Objective

• To develop a fast, end-to-end neural network for universal photorealistic style transfer that runs in real time on mobile devices.
• To enforce photorealism by learning local affine transforms that preserve edge structures and color consistency.
• To eliminate spatial grid artifacts common in grid-based style transfer methods.
• To ensure robustness and graceful degradation on out-of-distribution or adversarial inputs.
• To achieve high performance without post-processing or optimization at inference time.

Proposed method

• The model uses a feed-forward deep neural network to predict affine transforms in bilateral space, where each pixel's transformation is locally affine and edge-aware.
• The network learns to map input images to stylized outputs by predicting coefficients for a bilateral grid that represent local affine transforms.
• A bilateral-space Laplacian regularizer is introduced to suppress grid artifacts and improve spatial smoothness.
• The network is trained end-to-end on diverse image pairs, with the photorealism constraint enforced via local affine mapping in color space.
• Inference uses a low-resolution feature extraction head (256×256) for coefficient prediction, followed by hardware-accelerated trilinear interpolation on full resolution.
• The model is quantized to 16-bit floating point for efficient deployment on mobile GPUs.

Experimental results

Research questions

• RQ1 Can a feed-forward neural network achieve real-time photorealistic style transfer at 4K resolution on mobile hardware?
• RQ2 Can local affine transforms in bilateral space inherently enforce photorealism without optimization or post-processing?
• RQ3 How does the model perform on unseen or adversarial inputs compared to state-of-the-art methods?
• RQ4 Can the model generalize to video sequences while maintaining temporal coherence and style consistency?
• RQ5 What is the trade-off between photorealism, stylization fidelity, and visual quality compared to existing methods?

Key findings

• The method achieves real-time performance at 4K resolution on a mobile phone, running above 30 Hz with 16-bit quantization.
• The model is three orders of magnitude faster than state-of-the-art methods, with inference time nearly invariant to resolution.
• User studies show the method outperforms PhotoWCT, LST, and WCT 2 in overall image quality and stylization fidelity, with comparable photorealism to WCT 2.
• The model produces sharp, artifact-free results and degrades gracefully on unseen inputs such as portraits or monochromatic images.
• Video stylization is temporally coherent without additional regularization, demonstrating strong generalization to dynamic content.
• The bilateral-space Laplacian regularizer successfully eliminates spatial grid artifacts common in grid-based methods.