[Paper Review] A New Backpropagation Algorithm without Gradient Descent

Varun Ranganathan, Sayee Natarajan|arXiv (Cornell University)|Jan 25, 2018

Neural Networks and Applications5 references9 citations

TL;DR

This paper proposes a novel backpropagation algorithm that replaces gradient descent with Moore-Penrose pseudoinverse to compute weight and bias updates in artificial neural networks. By leveraging the pseudoinverse to directly solve for optimal parameters in a least-squares sense, the method avoids iterative gradient computation and works effectively with non-differentiable or ReLU-like activation functions, achieving up to 90.4% accuracy on the Wisconsin Breast Cancer dataset without relying on gradient-based optimization.

ABSTRACT

The backpropagation algorithm, which had been originally introduced in the 1970s, is the workhorse of learning in neural networks. This backpropagation algorithm makes use of the famous machine learning algorithm known as Gradient Descent, which is a first-order iterative optimization algorithm for finding the minimum of a function. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. In this paper, we develop an alternative to the backpropagation without the use of the Gradient Descent Algorithm, but instead we are going to devise a new algorithm to find the error in the weights and biases of an artificial neuron using Moore-Penrose Pseudo Inverse. The numerical studies and the experiments performed on various datasets are used to verify the working of this alternative algorithm.

Motivation & Objective

• To develop a backpropagation method that eliminates reliance on gradient descent for weight and bias updates in neural networks.
• To enable training of neural networks with non-differentiable or piecewise-linear activation functions by avoiding gradient computation.
• To improve training efficiency by replacing iterative gradient descent with a direct matrix-based solution using pseudoinverse.
• To validate the feasibility and performance of the proposed method on benchmark and real-world datasets.
• To explore the compatibility of the method with ReLU-like activation functions such as Softplus and LeakyReLU.

Proposed method

• The method computes weight and bias updates using the Moore-Penrose pseudoinverse of the input data matrix to solve for the optimal parameters that minimize the error in a least-squares sense.
• Each neuron is modified to have individual weights and biases per input, enabling a linear system formulation suitable for pseudoinverse computation.
• The algorithm assumes that the activation function is invertible over its output range, allowing the inverse to be applied to desired outputs to form target vectors for the pseudoinverse solution.
• The pseudoinverse is computed via singular value decomposition (SVD), ensuring a stable and unique solution even for rank-deficient or overdetermined systems.
• The method is applied layer-by-layer, treating each neuron’s input-output relationship as a linear system to be solved using the pseudoinverse.
• The approach avoids backpropagating gradients by directly solving for the parameter updates that would minimize the error, bypassing iterative optimization.

Experimental results

Research questions

• RQ1 Can a backpropagation algorithm be designed that does not rely on gradient descent for weight and bias updates?
• RQ2 Is the Moore-Penrose pseudoinverse a viable alternative to gradient descent for training artificial neural networks?
• RQ3 How well does the pseudoinverse-based method perform on non-differentiable or ReLU-like activation functions?
• RQ4 Can this method achieve competitive accuracy on real-world datasets without gradient-based optimization?
• RQ5 What are the limitations of this method in terms of activation function compatibility and convergence speed?

Key findings

• The proposed pseudoinverse-based backpropagation achieved 90.4% validation accuracy on the Wisconsin Breast Cancer dataset after 1000 epochs, demonstrating strong performance on real-world data.
• On the Two-Spirals problem, the method achieved approximately 63% accuracy, which is reasonable given the use of the Softplus activation function, which is suboptimal for this task.
• For the Concentric-Circles problem, the method reached 61% accuracy, again limited by the choice of Softplus activation, which is not ideal for circular data patterns.
• On the XOR problem, the method achieved 81% validation accuracy, showing that it can learn non-linear decision boundaries despite using a non-typical activation function.
• The training and validation errors stabilized after several hundred epochs, indicating convergence, though the error curves were erratic in early training phases.
• The method is most effective with activation functions whose inverse is defined over the full real line, such as Softplus, and is less effective with functions like Tanh due to domain restrictions.