[Paper Review] Weight Agnostic Neural Networks

Adam Gaier, David Ha|arXiv (Cornell University)|Jan 1, 2019

Neural Networks and Applications116 references77 citations

TL;DR

This paper introduces Weight Agnostic Neural Networks (WANNs), a method that discovers neural network architectures capable of solving tasks with randomly initialized weights, bypassing traditional training. By optimizing architectures for performance across a wide range of shared random weights, the approach finds minimal, high-performing architectures that achieve strong results—e.g., ~92% accuracy on MNIST and high rewards in continuous control tasks—demonstrating that architecture alone can encode effective solutions.

ABSTRACT

Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance. We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On a supervised learning domain, we find network architectures that achieve much higher than chance accuracy on MNIST using random weights. Interactive version of this paper at https://weightagnostic.github.io/

Motivation & Objective

• To investigate whether neural network architectures alone, without learning weights, can encode effective solutions for complex tasks.
• To develop a search method that prioritizes architectural inductive biases over weight optimization.
• To evaluate whether randomly initialized weights can yield high performance when paired with well-designed architectures.
• To demonstrate that minimal architectures can achieve strong performance across diverse tasks, including reinforcement learning and supervised learning.
• To encourage research into non-gradient-based learning methods by identifying architectures with intrinsic task-specific capabilities.

Proposed method

• Proposes a search framework that treats all weights as a single shared random parameter, sampled uniformly from a fixed range.
• Evaluates each architecture by averaging performance across multiple random weight values, optimizing for expected performance over this distribution.
• Uses a neuroevolution-based approach (inspired by NEAT) to evolve architectures with operations like adding nodes, connections, and changing activation functions.
• Employs tournament selection and mutation operators to explore architectural space while minimizing reliance on weight adaptation.
• Applies the search to both reinforcement learning tasks (Swing-Up CartPole, Bipedal Walker, Car Racing) and supervised learning (MNIST).
• Uses a fixed, shared weight value across all connections during evaluation to assess architecture robustness to weight initialization.

Experimental results

Research questions

• RQ1 Can neural network architectures alone, without any weight training, solve complex reinforcement learning tasks?
• RQ2 To what extent can randomly initialized weights still yield high performance when paired with a well-designed architecture?
• RQ3 Can minimal architectures be discovered that achieve performance significantly above chance on supervised learning tasks like MNIST?
• RQ4 How does the performance of weight-agnostic architectures compare to standard, trained networks on benchmark tasks?
• RQ5 What architectural inductive biases emerge from optimizing for performance across a wide range of random weights?

Key findings

• The method discovers minimal neural network architectures that achieve high performance on continuous control tasks like Bipedal Walker and Car Racing without any weight training, with average returns exceeding 300 and 500 respectively.
• On the MNIST dataset, WANNs achieve a test accuracy of approximately 92% using only random weights, significantly above the 10% chance level.
• The best-performing WANN architectures show robust performance across a wide range of shared weight values, with top networks maintaining high performance even when the shared weight is set to extreme values like ±2.
• Fine-tuning the shared weight parameter via policy gradient methods further improves performance, indicating that the architecture is a strong inductive bias that can be refined with minimal learning.
• The approach identifies architectures that outperform standard baselines in some tasks (e.g., Bipedal Walker) even when the baseline uses full weight training, suggesting strong inductive biases are encoded in the architecture.
• The method successfully discovers architectures that are both small and effective, with the MNIST WANN using only 5% of the connections of a standard network while achieving near-SOTA performance.