[论文解读] Exploring Randomly Wired Neural Networks for Image Recognition

Saining Xie, Alexander Kirillov+2 more|arXiv (Cornell University)|Apr 2, 2019

Advanced Neural Network Applications参考文献 50被引用 82

一句话总结

本论文研究由经典随机图模型（ER、BA、WS）生成的随机连线神经网络（RandWire），并展示它们在无需手工设计连线或操作搜索的情况下仍能达到具有竞争力的ImageNet准确率，强调网络生成器是设计中的关键要素。

ABSTRACT

Neural networks for image recognition have evolved through extensive manual design from simple chain-like models to structures with multiple wiring paths. The success of ResNets and DenseNets is due in large part to their innovative wiring plans. Now, neural architecture search (NAS) studies are exploring the joint optimization of wiring and operation types, however, the space of possible wirings is constrained and still driven by manual design despite being searched. In this paper, we explore a more diverse set of connectivity patterns through the lens of randomly wired neural networks. To do this, we first define the concept of a stochastic network generator that encapsulates the entire network generation process. Encapsulation provides a unified view of NAS and randomly wired networks. Then, we use three classical random graph models to generate randomly wired graphs for networks. The results are surprising: several variants of these random generators yield network instances that have competitive accuracy on the ImageNet benchmark. These results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design.

研究动机与目标

• Motivate and assess whether loosening manual wiring constraints via random graph generators can yield competitive image recognition models.
• Define a network generator framework and show how random graph models can instantiate neural networks.
• Evaluate RandWire networks on ImageNet across small, regular, and large computation regimes and compare to hand-designed and NAS architectures.

提出的方法

• Define the network generator g that maps a parameter set θ and a seed s to a neural network architecture.
• Generate complete neural networks by sampling random graphs from ER, BA, and WS models and mapping graphs to DAGs with node operations.
• Use a simple node operation design where each node aggregates inputs with positive weights and applies a ReLU-convolution-BN transformation, keeping FLOPs roughly independent of wiring.
• Construct networks in stages with input/output handling and progressive down-sampling similar to conventional CNNs (ResNet/DenseNet style).
• Train RandWire instances on ImageNet under fixed budgets, reporting mean accuracy and standard deviation across five seeds without performing per-generator random search.
• Compare RandWire to ResNet/ResNeXt and NAS-based models under multiple FLOP/parameter budgets.

实验结果

研究问题

• RQ1 Can randomly wired networks generated from classical random graph models achieve competitive ImageNet accuracy compared to hand-designed and NAS-designed architectures?
• RQ2 How does the choice of graph generator (ER, BA, WS) and its parameters influence accuracy and robustness of RandWire models?
• RQ3 What is the impact of node operation choices and network staging on performance when wiring is randomized?
• RQ4 How does RandWire performance scale across small, regular, and large computation regimes relative to conventional architectures?

主要发现

Network	Top-1 acc.	Top-5 acc.	FLOPs (M)	Params (M)
RandWire-WS (small regime)	74.7 ±0.25	92.2 ±0.15	583 ±6.2	5.6 ±0.1
ResNet-50	77.1	93.5	4.1	25.6
ResNeXt-50	78.4	94.0	4.2	25.0
RandWire-WS (109)	79.0 ±0.17	94.4 ±0.11	4.0 ±0.09	31.9 ±0.66
ResNet-101	78.8	94.4	7.8	44.6
ResNeXt-101	79.5	94.6	8.0	44.2
RandWire-WS (154)	80.1 ±0.19	94.8 ±0.18	7.9 ±0.18	61.5 ±1.32

• Randomly wired networks generated by ER, BA, and WS models achieve competitive ImageNet accuracy, with WS-based generators often matching or surpassing hand-designed counterparts under similar compute.
• WS-based RandWire variants can outperform or be on par with several NAS- and hand-designed networks while using comparable FLOPs and fewer parameters in some settings.
• Among random generators, the WS model with appropriate parameters (e.g., WS(4,0.75)) yields the best mean accuracy in the small computation regime (around 74.7% top-1).
• The accuracy variance across five seeds for a given generator is small (typical std around 0.2%–0.4%), while mean performance varies notably across generators, indicating the generator design encodes meaningful priors.
• Node operation choice (3x3 separable conv, regular conv, or pool-then-conv) has limited impact on the relative ranking of generators, suggesting wiring patterns contribute orthogonally to operation types.
• RandWire networks achieve competitive results in the regular computation regime (e.g., RandWire-WS with C=109/154) and can reach top-1 accuracies above 80% in larger models, while requiring fewer FLOPs and comparable parameters to ResNet/ResNeXt baselines.