[Paper Review] Slim, Sparse, and Shortcut Networks

Fenglei Fan, Dayang Wang+5 more|arXiv (Cornell University)|Nov 22, 2018

Domain Adaptation and Few-Shot Learning30 references4 citations

TL;DR

This paper proposes S3-Net, a slim, sparse, and shortcut-based neural network architecture that leverages theoretically grounded shortcut topologies to achieve universal approximation with minimal width. By demonstrating that a one-neuron-wide network with strategic shortcuts can approximate any univariate function, the authors establish a width-bounded universal approximator and show through spectral analysis and generalization bounds that shortcut connectivity enhances representational capacity and generalization, leading to superior performance over dense and state-of-the-art models on benchmarks.

ABSTRACT

Over the recent years, deep learning has become the mainstream data-driven approach to solve many real-world problems in many important areas. Among the successful network architectures, shortcut connections are well established to take the outputs of earlier layers as additional inputs to later layers, which have produced excellent results. Despite the extraordinary power, there remain important questions on the underlying mechanism and associated functionalities regarding shortcuts. For example, why are the shortcuts powerful? How to tune the shortcut topology to optimize the efficiency and capacity of the network model? Along this direction, here we first demonstrate a topology of shortcut connections that can make a one-neuron-wide deep network approximate any univariate function. Then, we present a novel width-bounded universal approximator in contrast to depth-bounded universal approximators. Next we demonstrate a family of theoretically equivalent networks, corroborated by the concerning statistical significance experiments, and their graph spectral characterization, thereby associating the representation ability of neural network with their graph spectral properties. Furthermore, we shed light on the effect of concatenation shortcuts on the margin-based multi-class generalization bound of deep networks. Encouraged by the positive results from the bounds analysis, we instantiate a slim, sparse, and shortcut network (S3-Net) and the experimental results demonstrate that the S3-Net can achieve better learning performance than the densely connected networks and other state-of-the-art models on some well-known benchmarks.

Motivation & Objective

• To understand the underlying mechanisms and functional roles of shortcut connections in deep neural networks.
• To develop a width-bounded universal approximator using shortcut topologies, contrasting with traditional depth-bounded approaches.
• To establish theoretical and empirical links between network topology, graph spectral properties, and representational capacity.
• To analyze the impact of concatenation shortcuts on margin-based multi-class generalization bounds.
• To design and evaluate S3-Net, a sparse, efficient architecture that outperforms dense and state-of-the-art models.

Proposed method

• Propose a novel one-neuron-wide deep network with strategically placed shortcut connections that can approximate any univariate function, establishing a width-bounded universal approximator.
• Formalize a family of theoretically equivalent neural networks and analyze their graph spectral properties to link topology to representational power.
• Derive and analyze margin-based generalization bounds for deep networks with concatenation shortcuts, showing improved generalization capacity.
• Design S3-Net as a practical instantiation of the theoretical findings, emphasizing sparsity, slim width, and shortcut integration.
• Use statistical significance tests and spectral characterization to validate the theoretical claims and robustness of the architecture.
• Evaluate S3-Net on well-known benchmarks and compare its performance against densely connected and state-of-the-art models.

Experimental results

Research questions

• RQ1 Why are shortcut connections powerful in deep neural networks?
• RQ2 How can shortcut topology be tuned to optimize model efficiency and capacity?
• RQ3 Can a one-neuron-wide deep network with shortcuts achieve universal approximation?
• RQ4 What is the relationship between graph spectral properties of a network and its representational ability?
• RQ5 How do concatenation shortcuts affect the margin-based generalization bound of deep networks?

Key findings

• A one-neuron-wide deep network with properly designed shortcut connections can universally approximate any univariate function, establishing a width-bounded universal approximator.
• Theoretical equivalence among a family of shortcut-connected networks is corroborated by statistical significance tests, supporting robustness of the architecture.
• Graph spectral analysis reveals a strong association between network topology and representational capacity, providing a new theoretical lens for understanding deep networks.
• Concatenation shortcuts are shown to improve the margin-based multi-class generalization bound, explaining their empirical success.
• S3-Net achieves better learning performance than densely connected networks and other state-of-the-art models on multiple well-known benchmarks.