[Paper Review] Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

Konstantin Schürholt, B. A. Knyazev+2 more|arXiv (Cornell University)|Sep 29, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a novel generative framework that uses hyper-representations—learned via self-supervised autoencoding on neural network weight populations—to sample high-performing, diverse neural network weights in a single forward pass. By introducing layer-wise loss normalization and topology-aware sampling strategies, the method generates models that outperform baselines in initialization, ensemble sampling, and transfer learning across multiple image datasets.

ABSTRACT

Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we extend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, performant and capable to outperform strong baselines as evaluated on several downstream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.

Motivation & Objective

• To enable the generation of novel, high-performing neural network weights from a model zoo without access to training data or labels.
• To address the challenge of generating functional models from hyper-representations by improving reconstruction quality and generalization.
• To develop sampling methods that leverage the geometric and topological structure of the hyper-representation space for diverse and effective weight generation.
• To demonstrate that hyper-representations can serve as a general-purpose generative model for neural network weights across downstream tasks.
• To enable conditional generation of weights on unseen architectures and latent factors without retraining the hyper-representation.

Proposed method

• Train a self-supervised autoencoder on flattened weight vectors from a model zoo to learn a low-dimensional hyper-representation using multi-head self-attention.
• Introduce layer-wise loss normalization in the reconstruction loss to stabilize training and improve decoded weight quality.
• Use a symmetric encoder-decoder architecture with learnable token embeddings and position encodings to model sequential neuron representations.
• Apply a multi-objective loss combining MSE and contrastive loss to enhance disentanglement and robustness in the latent space.
• Design sampling strategies based on the topology of the hyper-representation space, enabling diverse and high-performing weight generation in one forward pass.
• Condition sampling on latent factors or unseen architectures by interpolating or projecting within the learned hyper-representation space.

Experimental results

Research questions

• RQ1 Can hyper-representations trained on model zoos be effectively extended into a generative model for sampling novel, functional neural network weights?
• RQ2 How does layer-wise loss normalization improve the quality and performance of generated models compared to standard reconstruction losses?
• RQ3 To what extent can topology-aware sampling in the hyper-representation space generate diverse and high-performing models across different downstream tasks?
• RQ4 Can the proposed method generate models that outperform both random initialization and baselines in initialization, ensemble sampling, and transfer learning?
• RQ5 Can the hyper-representation be used to generate weights for unseen architectures without retraining, and how effective is this conditional generation?

Key findings

• Layer-wise loss normalization significantly improves the quality of decoded weights, enabling the generation of functional models that perform well on downstream tasks.
• The proposed sampling methods generate diverse model populations that achieve higher accuracy than random initialization and baseline hyper-representation methods.
• Generated models outperform strong baselines in initialization, ensemble sampling, and transfer learning, with performance gains observed across four image datasets.
• Even for unseen architectures, sampled models learn faster and achieve statistically significant performance improvements during fine-tuning and transfer learning.
• The method enables conditional generation of weights on latent factors and unseen architectures without retraining the hyper-representation, demonstrating generalization and flexibility.
• The approach achieves high performance with a single forward pass, making it efficient and scalable for practical deployment in model generation and knowledge distillation.