[Paper Review] Discovering Neural Wirings

Mitchell Wortsman, Ali Farhadi+1 more|arXiv (Cornell University)|Jan 1, 2019

Advanced Neural Network Applications36 citations

TL;DR

This paper proposes Deep Neural Wiring (DNW), a method that learns sparse, dynamic connectivity patterns in neural networks by allowing channels to form connections independently, enabling end-to-end structure and weight learning. It achieves 10% higher ImageNet accuracy than MobileNetV1 at ~41M FLOPs, outperforming handcrafted and randomly wired networks while generalizing to recurrent and continuous-time networks.

ABSTRACT

The success of neural networks has driven a shift in focus from feature engineering to architecture engineering. However, successful networks today are constructed using a small and manually defined set of building blocks. Even in methods of neural architecture search (NAS) the network connectivity patterns are largely constrained. In this work we propose a method for discovering neural wirings. We relax the typical notion of layers and instead enable channels to form connections independent of each other. This allows for a much larger space of possible networks. The wiring of our network is not fixed during training -- as we learn the network parameters we also learn the structure itself. Our experiments demonstrate that our learned connectivity outperforms hand engineered and randomly wired networks. By learning the connectivity of MobileNetV1we boost the ImageNet accuracy by 10% at ~41M FLOPs. Moreover, we show that our method generalizes to recurrent and continuous time networks. Our work may also be regarded as unifying core aspects of the neural architecture search problem with sparse neural network learning. As NAS becomes more fine grained, finding a good architecture is akin to finding a sparse subnetwork of the complete graph. Accordingly, DNW provides an effective mechanism for discovering sparse subnetworks of predefined architectures in a single training run. Though we only ever use a small percentage of the weights during the forward pass, we still play the so-called initialization lottery with a combinatorial number of subnetworks. Code and pretrained models are available at https://github.com/allenai/dnw while additional visualizations may be found at https://mitchellnw.github.io/blog/2019/dnw/.

Motivation & Objective

• To overcome the limitations of fixed architectural building blocks in neural networks by enabling dynamic, learnable connectivity.
• To address the constrained search space in neural architecture search (NAS) by relaxing layer-based constraints and enabling arbitrary channel connections.
• To unify neural architecture search with sparse neural network learning by discovering high-performing subnetworks through a single training run.
• To demonstrate that learning connectivity patterns leads to better performance than hand-designed or randomly wired architectures.
• To extend the method's applicability to recurrent and continuous-time networks beyond feedforward models.

Proposed method

• The method replaces traditional layers with a flexible wiring mechanism where each channel can connect to any other channel, forming a fully connected graph of potential connections.
• Connectivity is differentiable and trained jointly with network weights using a differentiable architecture search approach.
• A sparse routing mechanism is used to select only a small subset of connections during inference, reducing FLOPs while maintaining performance.
• The architecture is trained end-to-end, allowing both weights and connectivity patterns to be optimized simultaneously.
• The method treats architecture search as a combinatorial lottery, where the initialization lottery is played over a vast number of sparse subnetworks.
• The approach generalizes to recurrent and continuous-time networks by adapting the wiring mechanism to sequential and temporal computation.

Experimental results

Research questions

• RQ1 Can neural network connectivity be learned end-to-end rather than being fixed by architectural design?
• RQ2 Does learning sparse, dynamic wirings lead to better performance than handcrafted or randomly wired networks?
• RQ3 Can the method discover high-performing subnetworks of a complete graph architecture in a single training run?
• RQ4 How does the performance of learned wirings compare to standard architectures like MobileNetV1 on ImageNet?
• RQ5 Can the method be extended to recurrent and continuous-time neural networks?

Key findings

• The proposed method improves ImageNet top-1 accuracy by 10% over MobileNetV1 while using only ~41M FLOPs.
• The learned connectivity patterns outperform both hand-designed and randomly wired networks across multiple benchmarks.
• The method discovers high-performing sparse subnetworks from a complete graph architecture in a single training run, effectively solving a combinatorial architecture search problem.
• The approach generalizes to recurrent and continuous-time networks, demonstrating broad applicability beyond feedforward models.
• The method enables a form of the initialization lottery over a combinatorial number of sparse subnetworks, even though only a small fraction of weights are used during inference.
• Code and pretrained models are publicly available, supporting reproducibility and further research in sparse architecture discovery.