[论文解读] Generating Images with Perceptual Similarity Metrics based on Deep Networks

Alexey Dosovitskiy, Thomas Brox|arXiv (Cornell University)|Feb 8, 2016

Generative Adversarial Networks and Image Synthesis被引用 388

一句话总结

本论文提出 DeePSiM，一类在深度特征空间中衡量相似性并结合对抗与图像空间项的损失函数，以产生清晰、感知上真实的图像。它将该损失应用于自编码器、VAE，以及 AlexNet 表征的反演。

ABSTRACT

Image-generating machine learning models are typically trained with loss functions based on distance in the image space. This often leads to over-smoothed results. We propose a class of loss functions, which we call deep perceptual similarity metrics (DeePSiM), that mitigate this problem. Instead of computing distances in the image space, we compute distances between image features extracted by deep neural networks. This metric better reflects perceptually similarity of images and thus leads to better results. We show three applications: autoencoder training, a modification of a variational autoencoder, and inversion of deep convolutional networks. In all cases, the generated images look sharp and resemble natural images.

研究动机与目标

• 对于超越像素级损失而产生模糊的感知保真图像的需要进行动机说明。
• 提出一个损失类（DeePSiM），将特征空间、对抗与像素空间项结合起来。
• 展示三种实际应用：自编码器训练、一个 VAE 变体，以及深度视觉表示的反演。
• 证明与传统损失相比，DeePSiM 能产生更清晰、更自然的重建并保留细微结构。

提出的方法

• Define DeePSiM loss as L = lambda_feat * L_feat + lambda_adv * L_adv + lambda_img * L_img.
• L_feat measures distance in features: L_feat = sum_i ||C(G_theta(x_i)) - C(y_i)||_2^2 using a comparator C (e.g., AlexNet layers or Exemplar-CNN).
• L_adv uses a discriminator D_phi to impose a natural image prior via GAN-style adversarial training: L_discr = -sum_i log D_phi(y_i) + log(1 - D_phi(G_theta(x_i))); L_adv = - sum_i log D_phi(G_theta(x_i)).
• L_img is the image-space penalty: L_img = sum_i ||G_theta(x_i) - y_i||_2^2.
• Architectures include generators with up-convolution layers, three networks for data processing (generator G, discriminator D_phi, comparator C).
• Training uses Adam with specific hyperparameters and a strategy to stabilize adversarial training by balancing discriminator and adversarial losses.

实验结果

研究问题

• RQ1 Can deep feature-space losses better capture perceptual similarity for image generation than pixel-space losses?
• RQ2 How does combining feature loss with adversarial priors affect the realism and fidelity of generated images?
• RQ3 Do DeePSiM losses improve reconstruction quality in autoencoders, VAEs, and inversion of deep representations?
• RQ4 What choices of comparator (feature space) optimize performance for different tasks?
• RQ5 Is perceptual fidelity preserved across different layers when inverting deep networks?

主要发现

• DeePSiM-based autoencoders produce sharper, more textured reconstructions than SE or L1 losses, preserving fine structures.
• A VAE trained with DeePSiM produces images with more realistic statistics than standard pixel-space losses.
• Inversion of AlexNet representations using DeePSiM yields highly natural reconstructions, outperforming prior inversion methods in preserving perceptual details.
• Using a discriminator-based adversarial prior with a feature-space loss helps avoid overly blurry or chaotic reconstructions and yields more realistic images.
• Different feature spaces (e.g., AlexNet conv5, fc6, VideoNet) can be effective comparators, with AlexNet conv5 often providing best results, though other comparators still capture key image features.
• The combination of feature loss, adversarial loss, and image-space loss is superior to configurations that omit any component, indicating the necessity of all three terms for best performance.