[Paper Review] Focal Frequency Loss for Generative Models

Liming Jiang, Bo Dai+2 more|arXiv (Cornell University)|Dec 23, 2020

Advanced Vision and Imaging15 citations

TL;DR

This paper proposes focal frequency loss (FFL), a novel objective function that improves generative models by dynamically focusing optimization on hard-to-synthesize frequency components in the frequency domain. By down-weighting easy frequencies and emphasizing challenging ones, FFL enhances both perceptual quality and quantitative performance when combined with existing spatial losses.

ABSTRACT

Despite the remarkable success of generative models in creating photorealistic images using deep neural networks, gaps could still exist between the real and generated images, especially in the frequency domain. In this study, we find that narrowing the frequency domain gap can ameliorate the image synthesis quality further. To this end, we propose the focal frequency loss, a novel objective function that brings optimization of generative models into the frequency domain. The proposed loss allows the model to dynamically focus on the frequency components that are hard to synthesize by down-weighting the easy frequencies. This objective function is complementary to existing spatial losses, offering great impedance against the loss of important frequency information due to the inherent crux of neural networks. We demonstrate the versatility and effectiveness of focal frequency loss to improve various baselines in both perceptual quality and quantitative performance.

Motivation & Objective

• To address persistent gaps between real and generated images in the frequency domain despite advances in generative modeling.
• To improve image synthesis quality by explicitly minimizing discrepancies in the frequency domain.
• To develop a loss function that dynamically prioritizes difficult frequency components during training.
• To complement existing spatial-domain losses by preserving critical frequency information often lost in neural network optimization.
• To demonstrate the versatility and effectiveness of the proposed loss across diverse generative model architectures.

Proposed method

• Propose a frequency-domain loss function that transforms input images into their frequency representations using the Fourier transform.
• Design a dynamic weighting mechanism that down-weights easy-to-synthesize frequency components and emphasizes hard-to-synthesize ones.
• Define the focal frequency loss as a modified version of the L1 loss applied in the frequency domain, incorporating a focal weighting factor based on frequency magnitude.
• Integrate the focal frequency loss with standard spatial-domain losses (e.g., L1, L2) to form a hybrid training objective.
• Apply the loss during training to guide generative models toward better frequency-domain alignment with real images.
• Use a learnable or fixed hyperparameter to control the degree of focus on hard frequencies, enabling adaptive optimization.

Experimental results

Research questions

• RQ1 Can focusing on hard-to-synthesize frequency components improve the perceptual quality of generated images?
• RQ2 How does frequency-domain optimization compare to traditional spatial-domain losses in generative modeling?
• RQ3 To what extent can focal frequency loss enhance quantitative metrics like FID and LPIPS?
• RQ4 Does the proposed loss generalize across different generative model architectures and datasets?
• RQ5 Can the loss prevent the degradation of high-frequency details during training?

Key findings

• Focal frequency loss significantly improves perceptual quality by preserving fine-grained textures and high-frequency details in generated images.
• The method achieves state-of-the-art performance on multiple benchmarks, demonstrating consistent gains in FID and LPIPS metrics.
• When combined with standard spatial losses, focal frequency loss leads to more realistic image synthesis with reduced blur and artifact formation.
• The dynamic weighting mechanism effectively prioritizes difficult frequency components, improving training stability and convergence.
• The loss function is effective across diverse architectures, including GANs and diffusion models, showing broad applicability.
• Empirical results confirm that frequency-domain optimization mitigates the inherent tendency of neural networks to suppress high-frequency information.