[Paper Review] Sequential Training of Neural Networks with Gradient Boosting

Seyedsaman Emami, Gonzalo Martínez-Muñoz|arXiv (Cornell University)|Sep 26, 2019

Machine Learning and ELM4 citations

TL;DR

This paper proposes a novel sequential training method for neural networks using gradient boosting, where the final layers are trained iteratively by adding small subnetworks (one or a few neurons) at each step to minimize residuals. The approach achieves competitive generalization performance comparable to Adam and L-BFGS, with the added benefit of dynamic inference speed control by pruning later-trained neurons without significant accuracy loss.

ABSTRACT

This paper presents a novel technique based on gradient boosting to train the final layers of a neural network (NN). Gradient boosting is an additive expansion algorithm in which a series of models are trained sequentially to approximate a given function. A neural network can also be seen as an additive expansion where the scalar product of the responses of the last hidden layer and its weights provide the final output of the network. Instead of training the network as a whole, the proposed algorithm trains the network sequentially in $T$ steps. First, the bias term of the network is initialized with a constant approximation that minimizes the average loss of the data. Then, at each step, a portion of the network, composed of $J$ neurons, is trained to approximate the pseudo-residuals on the training data computed from the previous iterations. Finally, the $T$ partial models and bias are integrated as a single NN with $T imes J$ neurons in the hidden layer. Extensive experiments in classification and regression tasks, as well as in combination with deep neural networks, are carried out showing a competitive generalization performance with respect to neural networks trained with different standard solvers, such as Adam, L-BFGS, SGD and deep models. Furthermore, we show that the proposed method design permits to switch off a number of hidden units during test (the units that were last trained) without a significant reduction of its generalization ability. This permits the adaptation of the model to different classification speed requirements on the fly.

Motivation & Objective

• To develop a sequential training method for neural networks that improves generalization by leveraging gradient boosting principles.
• To enable dynamic model compression during inference by selectively deactivating the last-trained neurons.
• To provide an alternative to standard backpropagation solvers like Adam, SGD, and L-BFGS for training neural networks.
• To demonstrate that iterative training of small subnetworks can achieve performance competitive with full-network optimization.
• To explore the feasibility of using non-tree base estimators in gradient boosting frameworks for neural network training.

Proposed method

• The method initializes the network bias with a constant approximation that minimizes the average training loss.
• At each of T iterations, a small subnetwork with J neurons is trained to fit the pseudo-residuals from the previous iteration’s predictions.
• The subnetwork weights are updated using a Newton-Raphson step to minimize the loss function on the residuals.
• The residuals are then updated for the next iteration, and the process continues until convergence.
• The final model is a single neural network with T×J hidden units, combining all sequentially trained subnetworks.
• During inference, neurons trained last can be deactivated on the fly to reduce computational cost with minimal accuracy drop.

Experimental results

Research questions

• RQ1 Can gradient boosting be effectively adapted to train the final layers of a neural network in a sequential, additive manner?
• RQ2 Does this sequential training approach achieve generalization performance competitive with standard solvers like Adam and L-BFGS?
• RQ3 Can the model be dynamically pruned during inference by removing later-trained neurons without significant performance degradation?
• RQ4 How does the performance of the proposed method compare across classification and regression tasks, especially in tabular data settings?
• RQ5 Can this method be integrated with deep neural networks in transfer learning scenarios?

Key findings

• The proposed method achieves generalization performance competitive with Adam and L-BFGS across multiple classification and regression tasks.
• In regression tasks, the method achieved the best performance on most evaluated datasets, outperforming other solvers.
• The model's generalization accuracy converges with the number of trained subnetworks (or hidden units), indicating stable learning dynamics.
• The method enables on-the-fly model compression: deactivating the last-trained neurons results in minimal performance drop, allowing runtime adaptation to computational constraints.
• The approach is effective when combined with deep neural networks in transfer learning setups, showing robustness across different architectures.
• The method demonstrates that iterative training of small, focused subnetworks can yield strong performance, even surpassing standard backpropagation in some regression scenarios.