[Paper Review] Neural Network Architecture Beyond Width and Depth

Zuowei Shen, Haizhao Yang+1 more|arXiv (Cornell University)|May 19, 2022

Neural Networks and Applications4 citations

TL;DR

This paper introduces a novel three-dimensional neural network architecture—NestNet—by adding a new hyperparameter, height, beyond traditional width and depth. By recursively composing sub-networks as activation functions, NestNets achieve superior approximation power: with O(n) parameters, a height-s NestNet approximates 1-Lipschitz functions on [0,1]^d with error O(n^{-(s+1)/d}), outperforming standard ReLU networks, which achieve only O(n^{-2/d}) under the same parameter budget.

ABSTRACT

This paper proposes a new neural network architecture by introducing an additional dimension called height beyond width and depth. Neural network architectures with height, width, and depth as hyper-parameters are called three-dimensional architectures. It is shown that neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures (those with only width and depth as hyper-parameters), e.g., standard fully connected networks. The new network architecture is constructed recursively via a nested structure, and hence we call a network with the new architecture nested network (NestNet). A NestNet of height $s$ is built with each hidden neuron activated by a NestNet of height $\le s-1$. When $s=1$, a NestNet degenerates to a standard network with a two-dimensional architecture. It is proved by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate $1$-Lipschitz continuous functions on $[0,1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, while the optimal approximation error of standard ReLU networks with $\mathcal{O}(n)$ parameters is $\mathcal{O}(n^{-2/d})$. Furthermore, such a result is extended to generic continuous functions on $[0,1]^d$ with the approximation error characterized by the modulus of continuity. Finally, we use numerical experimentation to show the advantages of the super-approximation power of ReLU NestNets.

Motivation & Objective

• To address the theoretical and practical limitations of standard two-dimensional neural networks (width and depth only), which have reached near-optimal approximation rates.
• To propose a new architectural paradigm that introduces a third hyperparameter—height—to significantly enhance expressive capacity.
• To demonstrate that recursive nesting of sub-networks as activation functions enables super-approximation power beyond standard networks.
• To validate the theoretical advantages through numerical experiments on image classification tasks.

Proposed method

• Proposes a three-dimensional architecture where neural networks are defined by height, width, and depth as hyperparameters, extending beyond standard two-dimensional designs.
• Introduces the Nested Network (NestNet) as a recursive architecture: a height-s NestNet uses sub-networks of height ≤ s−1 as activation functions for its neurons.
• Employs ReLU activation functions and constructs the network via nested composition, enabling parameter sharing across sub-networks.
• Derives theoretical approximation error bounds using the modulus of continuity and applies them to 1-Lipschitz and general continuous functions on [0,1]^d.
• Designs a modified CNN architecture (CNN2) where selected ReLU activations are replaced by sub-networks (NestNet of height 2), enabling comparison with standard CNN (CNN1).
• Employs RAdam optimizer with learning rate scheduling, batch normalization, and dropout to stabilize training, and evaluates performance on Fashion-MNIST.

Experimental results

Research questions

• RQ1 Can introducing a new architectural dimension—height—beyond width and depth lead to a significant improvement in approximation power?
• RQ2 How does the approximation error of a height-s ReLU NestNet scale with the number of parameters, compared to standard ReLU networks?
• RQ3 Can the recursive nesting of sub-networks as activation functions lead to better generalization and performance in practical deep learning tasks?
• RQ4 What is the theoretical relationship between the height of a NestNet and its ability to approximate continuous functions with a given modulus of continuity?

Key findings

• A height-s ReLU NestNet with O(n) parameters achieves an approximation error of O(n^{-(s+1)/d}) for 1-Lipschitz functions on [0,1]^d, significantly better than the O(n^{-2/d}) error of standard ReLU networks.
• For generic continuous functions, the approximation error of a height-s NestNet is characterized by the modulus of continuity, demonstrating improved convergence rates with increasing height.
• Numerical experiments on Fashion-MNIST show that a NestNet-based CNN (CNN2) achieves higher test accuracy (0.9260 average in last 100 epochs) than a standard CNN (CNN1) with 0.9244, despite slightly longer training time and 10 additional parameters.
• The best-performing trial of CNN2 reached 92.66% test accuracy, outperforming CNN1’s 92.53% maximum accuracy, confirming the practical advantage of the NestNet architecture.
• The theoretical and empirical results together demonstrate that NestNets provide a super-approximation power due to recursive parameter sharing via nested sub-networks.