[Paper Review] Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks

Andrea Ceni|arXiv (Cornell University)|Oct 3, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Random Orthogonal Additive Filters (roaFNN and roaRNN), a novel deep neural network architecture that mathematically prevents vanishing or exploding gradients by orthogonal filtering and residual-like addition of activations. The method ensures stable gradient flow via analytical bounds on the input-output Jacobian, enabling training of 50,000-layer feedforward networks and 10,000-step recurrent sequences with superfast convergence and state-of-the-art performance.

ABSTRACT

Since the recognition in the early nineties of the vanishing/exploding (V/E) gradient issue plaguing the training of neural networks (NNs), significant efforts have been exerted to overcome this obstacle. However, a clear solution to the V/E issue remained elusive so far. In this manuscript a new architecture of NN is proposed, designed to mathematically prevent the V/E issue to occur. The pursuit of approximate dynamical isometry, i.e. parameter configurations where the singular values of the input-output Jacobian are tightly distributed around 1, leads to the derivation of a NN's architecture that shares common traits with the popular Residual Network model. Instead of skipping connections between layers, the idea is to filter the previous activations orthogonally and add them to the nonlinear activations of the next layer, realising a convex combination between them. Remarkably, the impossibility for the gradient updates to either vanish or explode is demonstrated with analytical bounds that hold even in the infinite depth case. The effectiveness of this method is empirically proved by means of training via backpropagation an extremely deep multilayer perceptron of 50k layers, and an Elman NN to learn long-term dependencies in the input of 10k time steps in the past. Compared with other architectures specifically devised to deal with the V/E problem, e.g. LSTMs for recurrent NNs, the proposed model is way simpler yet more effective. Surprisingly, a single layer vanilla RNN can be enhanced to reach state of the art performance, while converging super fast; for instance on the psMNIST task, it is possible to get test accuracy of over 94% in the first epoch, and over 98% after just 10 epochs.

Motivation & Objective

• To solve the long-standing vanishing/exploding gradient problem in deep neural networks, which has plagued training since the 1990s.
• To design a deep network architecture that analytically guarantees stable gradient dynamics, even in infinite depth.
• To achieve this stability without relying on heuristics like batch normalization or gradient clipping.
• To demonstrate the effectiveness of the architecture on extremely deep feedforward and recurrent networks.
• To provide a theoretically grounded, simple alternative to complex architectures like ResNets or LSTMs.

Proposed method

• Proposes a new feedforward network (roaFNN) where each layer computes a convex combination of nonlinear activations and orthogonally filtered previous activations.
• Uses random semi-orthogonal matrices $O_l$ to filter and add previous layer outputs, ensuring isometric properties in the input-output Jacobian.
• Introduces a learnable scalar hyperparameter $\rho$ to control the balance between nonlinear and filtered components.
• Derives analytical bounds on the maximum singular value of the input-output Jacobian, proving gradient stability under mild conditions.
• Applies the same principle to recurrent networks (roaRNN), enabling training through time for long-term dependency tasks.
• Employs matrix norm inequalities and the isometric property of semi-orthogonal matrices to prove that the Jacobian’s norm remains bounded.

Experimental results

Research questions

• RQ1 Can a deep neural network architecture be designed such that vanishing or exploding gradients are mathematically impossible, even in infinite depth?
• RQ2 How can orthogonal filtering of previous activations stabilize gradient dynamics in deep networks?
• RQ3 Can a simple, gateless architecture outperform complex models like LSTMs or ResNets in long-term sequence learning?
• RQ4 What is the impact of random orthogonal matrices versus structured ones (e.g., identity) on training stability and performance?
• RQ5 Does the use of orthogonal additive filters implicitly regularize the loss landscape and improve generalization?

Key findings

• The proposed roaFNN architecture prevents vanishing and exploding gradients through analytical bounds on the input-output Jacobian, even in the infinite depth limit.
• A 50,000-layer feedforward network was successfully trained via backpropagation, demonstrating the method’s scalability.
• The roaRNN model achieved over 94% test accuracy on psMNIST in just one epoch and over 98% after ten epochs, outperforming standard RNNs and LSTMs.
• A single-layer vanilla RNN enhanced with the roaRNN mechanism reached state-of-the-art performance on long-term dependency tasks.
• The method’s simplicity and effectiveness surpass more complex architectures like LSTMs and ResNets, which rely on trainable gates or skip connections.
• Empirical results suggest that random orthogonal filters may implicitly regularize the loss landscape, improving training dynamics.