[Paper Review] Focal Frequency Loss for Image Reconstruction and Synthesis

Liming Jiang, Bo Dai+2 more|arXiv (Cornell University)|Dec 23, 2020

Advanced Vision and Imaging78 references29 citations

TL;DR

This paper proposes focal frequency loss (FFL), a novel frequency-domain loss function that adaptively emphasizes hard-to-synthesize high-frequency components during image generation by down-weighting easy frequencies via a dynamic spectrum weight matrix. FFL improves perceptual quality and quantitative metrics across diverse models—including VAE, pix2pix, SPADE, and StyleGAN2—by narrowing the frequency domain gap between real and generated images.

ABSTRACT

Image reconstruction and synthesis have witnessed remarkable progress thanks to the development of generative models. Nonetheless, gaps could still exist between the real and generated images, especially in the frequency domain. In this study, we show that narrowing gaps in the frequency domain can ameliorate image reconstruction and synthesis quality further. We propose a novel focal frequency loss, which allows a model to adaptively focus on frequency components that are hard to synthesize by down-weighting the easy ones. This objective function is complementary to existing spatial losses, offering great impedance against the loss of important frequency information due to the inherent bias of neural networks. We demonstrate the versatility and effectiveness of focal frequency loss to improve popular models, such as VAE, pix2pix, and SPADE, in both perceptual quality and quantitative performance. We further show its potential on StyleGAN2.

Motivation & Objective

• Address the persistent gap between real and generated images in the frequency domain, particularly the loss of high-frequency details and artifacts like checkerboard patterns.
• Overcome the spectral bias of neural networks, which favor learning low-frequency components and neglect hard-to-synthesize high-frequency components.
• Develop a frequency-domain loss function that enables models to adaptively focus on difficult frequency components during training.
• Improve image reconstruction and synthesis quality by directly optimizing frequency representations, complementing existing spatial-domain losses.
• Demonstrate the generalizability and effectiveness of FFL across diverse architectures, including autoencoders, GANs, and style-based generators.

Proposed method

• Transform input and generated images into their frequency representations using the discrete Fourier transform (DFT), capturing both amplitude and phase information.
• Represent each frequency component as a 2D vector combining magnitude and phase, enabling joint optimization in the frequency domain.
• Define a scaled Euclidean distance between frequency vectors of real and generated images to measure spectral discrepancy.
• Introduce a dynamic spectrum weight matrix that down-weights easy frequencies (low loss) and up-weights hard frequencies (high loss) during training.
• Apply focal loss-style weighting to the frequency domain loss, enabling adaptive focus on challenging frequency components through a non-uniform distribution.
• Integrate FFL as a complementary objective with existing spatial losses (e.g., perceptual loss, L1/L2 loss) to enhance overall training stability and quality.

Experimental results

Research questions

• RQ1 Can optimizing in the frequency domain improve image reconstruction and synthesis quality beyond spatial-domain losses?
• RQ2 To what extent does spectral bias in neural networks hinder the learning of high-frequency components in generated images?
• RQ3 Can a frequency-aware loss function that adaptively emphasizes hard frequencies lead to perceptually superior and quantitatively better results?
• RQ4 How does FFL perform across diverse architectures, including VAEs, pix2pix, SPADE, and StyleGAN2?
• RQ5 Does FFL effectively reduce periodic artifacts and spectral distortions commonly found in GAN-generated images?

Key findings

• FFL significantly improves FID scores: on CelebA-HQ (1024×1024), StyleGAN2 with FFL achieves a FID of 3.374, outperforming the original model’s 3.733.
• In image-to-image translation (edges→shoes), FFL reduces FID from 80.279 (baseline) to 74.359, with IS improving from 2.674 to 2.804.
• On anime portraits (64×64), FFL boosts PSNR from 19.885 to 20.657, SSIM from 0.575 to 0.628, and reduces LFD from 14.822 to 14.644.
• Visual analysis confirms that FFL narrows the frequency domain gap: VAEs trained with FFL no longer bias toward limited spectrum regions and recover high-frequency details.
• FFL enables the generation of essential spectral patterns (e.g., periodic structures) that are lost in baseline models, indicating improved frequency fidelity.
• Even without truncation, StyleGAN2 with FFL produces photorealistic images with fewer artifacts (e.g., on eyes and teeth), confirming improved detail synthesis.