[Paper Review] Style-Aware Normalized Loss for Improving Arbitrary Style Transfer

Jiaxin Cheng, Ayush Jaiswal+3 more|arXiv (Cornell University)|Apr 18, 2021

Generative Adversarial Networks and Image Synthesis50 references4 citations

TL;DR

This paper proposes a style-aware normalized loss to address imbalanced style transferability (IST) in arbitrary style transfer (AST), where models under- or over-stylize images due to equal weighting of style losses during training. By deriving theoretical bounds of the style loss and introducing style-aware normalization, the method achieves 111% relative improvement in deception rate and 98% higher human preference for stylized outputs across four AST models.

ABSTRACT

Neural Style Transfer (NST) has quickly evolved from single-style to infinite-style models, also known as Arbitrary Style Transfer (AST). Although appealing results have been widely reported in literature, our empirical studies on four well-known AST approaches (GoogleMagenta, AdaIN, LinearTransfer, and SANet) show that more than 50% of the time, AST stylized images are not acceptable to human users, typically due to under- or over-stylization. We systematically study the cause of this imbalanced style transferability (IST) and propose a simple yet effective solution to mitigate this issue. Our studies show that the IST issue is related to the conventional AST style loss, and reveal that the root cause is the equal weightage of training samples irrespective of the properties of their corresponding style images, which biases the model towards certain styles. Through investigation of the theoretical bounds of the AST style loss, we propose a new loss that largely overcomes IST. Theoretical analysis and experimental results validate the effectiveness of our loss, with over 80% relative improvement in style deception rate and 98% relatively higher preference in human evaluation.

Motivation & Objective

• To investigate the root cause of imbalanced style transferability (IST) in arbitrary style transfer (AST) models.
• To identify that equal sample-wise loss weighting across diverse styles biases models toward certain styles.
• To develop a theoretically grounded, style-aware loss function that normalizes training loss based on style-specific properties.
• To validate the new loss through extensive benchmarking and human evaluation across four state-of-the-art AST models.
• To improve alignment between loss optimization and human perception of stylization quality.

Proposed method

• The authors analyze the theoretical bounds of the conventional Gram matrix-based style loss and derive its expected value under varying style characteristics.
• They propose a new style-aware normalized loss that reweights sample-wise style losses based on the intrinsic properties of each style image, ensuring balanced contribution during training.
• The method normalizes the style loss per sample using a learned or estimated scale factor derived from the style image's feature statistics.
• The new loss is integrated into existing AST frameworks without architectural changes, enabling plug-and-play improvement.
• Theoretical analysis shows the new loss is positively correlated with human perception, unlike the classic loss.
• The approach is evaluated using deception rate and human preference studies on ImageNet and PBN/DTD datasets.

Figure 2 : Distribution of classic Gram matrix-based style losses for four AST methods [ 14 , 19 , 29 , 37 ] . Smaller loss does not guarantee better style transfer (left two images) while high quality transferred images can have larger style losses (middle two images), with over-stylized images cou

Experimental results

Research questions

• RQ1 Why do state-of-the-art AST models fail to generalize well across diverse styles, often producing under- or over-stylized outputs?
• RQ2 How does the conventional style loss fail to reflect human perception of stylization quality?
• RQ3 What causes the imbalance in style transferability (IST) across different style images during training?
• RQ4 Can a theoretically grounded, style-aware normalization of the style loss improve training balance and output quality?
• RQ5 Is the proposed loss effective across multiple AST models and metrics like deception rate and human preference?

Key findings

• The proposed style-aware normalized loss achieves a 111% relative improvement in deception rate across all tested AST models, with LinearTransfer showing the highest gain of 111%.
• Human evaluation shows 55.9% preference for images stylized with the new loss, a 98% relative increase over the classic loss.
• The new loss is positively correlated with human perception, unlike the classic loss, which fails to reflect human evaluation scores.
• The method effectively mitigates under-stylization and over-stylization issues in AST, reducing unacceptable outputs to under 50% across models.
• Theoretical analysis confirms that the new loss better reflects the true style transfer quality by accounting for style-specific variance in feature distributions.
• The loss is universally applicable and improves performance across four distinct AST models: GoogleMagenta, AdaIN, LinearTransfer, and SANet.

Figure 3 : Statistics of human perception of stylization quality as assessed in Study II.