[Paper Review] On Expressivity and Trainability of Quadratic Networks

Fenglei Fan, Mengzhou Li+3 more|arXiv (Cornell University)|Oct 12, 2021

Model Reduction and Neural Networks39 references4 citations

TL;DR

This paper establishes the superior expressivity of quadratic neural networks over conventional and ReLU-activated networks using spline theory and algebraic geometry. It introduces ReLinear, a novel training strategy that stabilizes training by reinitializing quadratic weights, significantly improving performance on image classification and depth estimation benchmarks like CIFAR-10 and KITTI.

ABSTRACT

Inspired by the diversity of biological neurons, quadratic artificial neurons can play an important role in deep learning models. The type of quadratic neurons of our interest replaces the inner-product operation in the conventional neuron with a quadratic function. Despite promising results so far achieved by networks of quadratic neurons, there are important issues not well addressed. Theoretically, the superior expressivity of a quadratic network over either a conventional network or a conventional network via quadratic activation is not fully elucidated, which makes the use of quadratic networks not well grounded. Practically, although a quadratic network can be trained via generic backpropagation, it can be subject to a higher risk of collapse than the conventional counterpart. To address these issues, we first apply the spline theory and a measure from algebraic geometry to give two theorems that demonstrate better model expressivity of a quadratic network than the conventional counterpart with or without quadratic activation. Then, we propose an effective training strategy referred to as ReLinear to stabilize the training process of a quadratic network, thereby unleashing the full potential in its associated machine learning tasks. Comprehensive experiments on popular datasets are performed to support our findings and confirm the performance of quadratic deep learning. We have shared our code in \url{https://github.com/FengleiFan/ReLinear}.

Motivation & Objective

• To theoretically demonstrate that quadratic networks have higher model expressivity than conventional networks and ReLU-activated conventional networks.
• To address the instability and risk of collapse during training in quadratic networks due to high-degree polynomial outputs.
• To develop a practical and effective training strategy that enables stable optimization of quadratic networks.
• To empirically validate the theoretical claims and the effectiveness of the proposed method on diverse machine learning tasks.
• To show that replacing conventional neurons with quadratic neurons in existing models (e.g., LapDepth) yields measurable performance gains.

Proposed method

• Uses spline theory to prove that quadratic networks can represent functions with higher expressivity than conventional networks under the same architecture.
• Applies a measure from algebraic geometry to compare expressivity between quadratic networks and ReLU-activated conventional networks.
• Proposes ReLinear, a reinitialization strategy that resets the quadratic weight matrices to zero and reinitializes linear weights to stabilize training.
• Reinitializes quadratic weights to zero and uses Kaiming initialization for linear weights to reduce gradient explosion and output oscillation.
• Employs standard deep learning training protocols (AdamW, learning rate decay, batch normalization) with ReLinear applied only to quadratic layers.
• Validates the method on image classification (CIFAR-10, CIFAR-100) and depth estimation (KITTI), replacing conventional neurons in ResNeXt and LapDepth with quadratic neurons.

Figure 1: Illustration of why oscillation occurs in quadratic networks. The network structure is 1-10-10-10-10-1, and the activation function is ReLU. All parameters of the quadratic and conventional networks are initialized with the normal distribution $\mathcal{N}(0,\sigma^{2})$ , where $\sigma$ i

Experimental results

Research questions

• RQ1 Can quadratic networks express functions that conventional networks or ReLU-activated conventional networks cannot, even with identical architecture?
• RQ2 What theoretical mechanisms underlie the superior expressivity of quadratic networks compared to conventional and ReLU-activated models?
• RQ3 Why do quadratic networks exhibit higher training instability and risk of collapse compared to conventional networks?
• RQ4 Can a simple reinitialization strategy like ReLinear effectively stabilize the training of quadratic networks without architectural changes?
• RQ5 Does replacing conventional neurons with quadratic neurons in existing models lead to measurable performance improvements on real-world tasks?

Key findings

• Theoretical analysis using spline theory and algebraic geometry confirms that quadratic networks have higher expressivity than conventional networks and ReLU-activated conventional networks.
• Quadratic networks can represent certain functions with polynomial-sized architectures, while conventional networks require exponentially larger architectures for the same approximation.
• A univariate ablation shows that small changes in weight initialization variance cause massive output variations in quadratic networks, explaining their training instability.
• On CIFAR-10, the ReLinear-regularized quadratic network achieves 95.8% test accuracy, outperforming the baseline conventional network.
• On the KITTI depth estimation benchmark, Quadratic-LapDepth reduces Sq Rel from 0.214 (baseline LapDepth) to 0.200 and RMSE from 0.458 to 0.427.
• The proposed ReLinear strategy effectively stabilizes training and enables quadratic networks to achieve state-of-the-art performance on multiple tasks without architectural modifications.

Figure 2: The performance of a quadratic network trained using the proposed ReLinear method, with an observed improvement than the conventional network of the same structure. $(\gamma_{g},\gamma_{b})$ , $(\alpha_{g},\alpha_{b})$ , and $(\beta_{g},\beta_{b})$ are hyperparameters of ReLinear . As thes