[Paper Review] Generative Hierarchical Features from Synthesizing Images

Yinghao Xu, Yujun Shen+3 more|arXiv (Cornell University)|Jul 20, 2020

Generative Adversarial Networks and Image Synthesis72 references4 citations

TL;DR

This paper proposes Generative Hierarchical Features (GH-Feat), a novel method that leverages a pre-trained StyleGAN generator as a learned loss function to train a hierarchical encoder, producing highly transferable visual features. The approach achieves state-of-the-art performance in both generative and discriminative tasks, including image editing, harmonization, classification, and landmark detection, outperforming existing GAN-based methods in reconstruction quality and feature transferability.

ABSTRACT

Generative Adversarial Networks (GANs) have recently advanced image synthesis by learning the underlying distribution of the observed data. However, how the features learned from solving the task of image generation are applicable to other vision tasks remains seldom explored. In this work, we show that learning to synthesize images can bring remarkable hierarchical visual features that are generalizable across a wide range of applications. Specifically, we consider the pre-trained StyleGAN generator as a learned loss function and utilize its layer-wise representation to train a novel hierarchical encoder. The visual feature produced by our encoder, termed as Generative Hierarchical Feature (GH-Feat), has strong transferability to both generative and discriminative tasks, including image editing, image harmonization, image classification, face verification, landmark detection, and layout prediction. Extensive qualitative and quantitative experimental results demonstrate the appealing performance of GH-Feat.

Motivation & Objective

• To investigate whether features learned from image generation via GANs can be effectively repurposed for diverse vision tasks beyond classification.
• To address the limitation of existing GAN-based features that are often restricted to high-level classification and lack hierarchical, transferable representations.
• To develop a hierarchical encoder that aligns with the layer-wise features of a pre-trained StyleGAN generator for improved feature extraction.
• To evaluate the generalization capability of these features across both generative and discriminative downstream tasks.

Proposed method

• The pre-trained StyleGAN generator is treated as a learned loss function, replacing traditional reconstruction losses in encoder training.
• A hierarchical encoder is trained to produce feature maps that match the layer-wise activations of the StyleGAN generator when fed the same input image.
• The encoder is trained using adversarial and perceptual supervision from the generator, with the generator fixed after pre-training on ImageNet.
• Feature extraction is performed at the 'res 5' stage of the encoder, followed by adaptive average pooling and vectorization for downstream classification.
• The method enables end-to-end training of the encoder using the generator’s internal representations as supervisory signals.
• The approach supports both image reconstruction and style mixing, demonstrating hierarchical disentanglement of content and style.

Experimental results

Research questions

• RQ1 Can a pre-trained GAN generator serve as an effective learned loss function for training a general-purpose visual feature encoder?
• RQ2 To what extent do features extracted via this method transfer across diverse vision tasks, including generative and discriminative applications?
• RQ3 How does the hierarchical structure of the encoder compare to existing methods in preserving identity and texture during style transfer?
• RQ4 Does reusing a pre-trained generator yield better performance and efficiency than training the generator and encoder jointly from scratch?

Key findings

• GH-Feat achieves a Fréchet Inception Distance (FID) of 18.48 on ImageNet image reconstruction, significantly outperforming the baseline method that trains generator and encoder from scratch (FID: 46.20).
• The method achieves a mean squared error (MSE) of 0.0464 and a structural similarity (SSIM) of 0.558, demonstrating superior reconstruction quality.
• In style mixing tasks, GH-Feat preserves identity and pose from style images better than ALAE, especially at high-level feature mixing.
• GH-Feat achieves competitive performance in image classification, matching state-of-the-art methods like BigBiGAN while being more efficient and generalizable.
• The ablation study confirms that reusing a pre-trained generator leads to better performance and faster convergence than joint training from scratch.
• The hierarchical features show strong transferability across tasks, including face verification, landmark detection, and layout prediction, with consistent improvements over prior GAN-based feature learning methods.