[Paper Review] GrCAN: Gradient Boost Convolutional Autoencoder with Neural Decision Forest

Manqing Dong, Lina Yao+3 more|arXiv (Cornell University)|Jun 21, 2018

Advanced Neural Network ApplicationsComputer Science42 references4 citations

TL;DR

GrCAN proposes an end-to-end differentiable model that integrates a convolutional autoencoder, a neural decision forest, and gradient boosting to enhance classification performance. By jointly training the autoencoder for feature learning, the neural decision forest for decision-making, and gradient boost modules for residual learning, the model achieves state-of-the-art accuracy on diverse datasets, including Fashion-MNIST with over 92% accuracy.

ABSTRACT

Random forest and deep neural network are two schools of effective classification methods in machine learning. While the random forest is robust irrespective of the data domain, the deep neural network has advantages in handling high dimensional data. In view that a differentiable neural decision forest can be added to the neural network to fully exploit the benefits of both models, in our work, we further combine convolutional autoencoder with neural decision forest, where autoencoder has its advantages in finding the hidden representations of the input data. We develop a gradient boost module and embed it into the proposed convolutional autoencoder with neural decision forest to improve the performance. The idea of gradient boost is to learn and use the residual in the prediction. In addition, we design a structure to learn the parameters of the neural decision forest and gradient boost module at contiguous steps. The extensive experiments on several public datasets demonstrate that our proposed model achieves good efficiency and prediction performance compared with a series of baseline methods.

Motivation & Objective

• To address the limitations of traditional random forests and deep neural networks by unifying their strengths in a single differentiable model.
• To improve classification performance on both small and high-dimensional datasets by integrating gradient boosting into the neural decision forest framework.
• To enhance feature representation learning by combining convolutional autoencoders with neural decision forests for better generalization.
• To develop a trainable, end-to-end architecture that jointly optimizes autoencoder, neural decision forest, and gradient boost components.
• To evaluate the model's robustness and scalability across diverse datasets, including image and tabular data.

Proposed method

• Employs a convolutional autoencoder to learn hierarchical, low-dimensional representations of input data, capturing local spatial relationships.
• Uses a neural decision forest as a differentiable classifier, where decision nodes are parameterized as probability distributions for end-to-end training.
• Introduces a gradient boost module that learns residuals between predictions and ground-truth labels, improving model accuracy iteratively.
• Designs a contiguous training strategy that alternates between updating the autoencoder, neural decision forest, and gradient boost modules in a unified optimization loop.
• Enables extendable gradient boost modules, allowing multiple boosting stages to refine predictions progressively.
• Applies min-max normalization to input features and uses a batch size of 200 during training for stable optimization.

Experimental results

Research questions

• RQ1 Can gradient boosting be effectively integrated into a differentiable neural decision forest to improve classification accuracy?
• RQ2 How does combining a convolutional autoencoder with a neural decision forest enhance feature representation and model performance?
• RQ3 What is the impact of varying hyperparameters such as activation functions, autoencoder depth, and number of trees on model performance?
• RQ4 Does the proposed end-to-end training strategy with joint optimization of autoencoder, neural decision forest, and gradient boost modules yield better results than separate training?
• RQ5 How does the model perform across diverse datasets of varying sizes and data types, including image and tabular data?

Key findings

• The proposed GrCAN model achieves over 92% accuracy on the Fashion-MNIST dataset, outperforming baseline methods and state-of-the-art models.
• Models with one or two gradient boost modules consistently outperform those without gradient boosting, demonstrating the effectiveness of residual learning in the neural decision forest framework.
• A two-layer convolutional autoencoder with sigmoid activation achieved the best balance between accuracy and training efficiency, outperforming one- and three-layer variants in convergence speed and final performance.
• The number of trees and tree depth in the neural decision forest significantly influence accuracy, with higher values leading to faster performance gains, especially in later training epochs.
• The model converges more slowly with deeper autoencoders and larger forests, but the increased complexity leads to better generalization and higher final accuracy.
• The learning rate hyperparameter ρ in the gradient boost module has a notable impact: higher values like ρ=1.0 required more iterations but achieved superior final accuracy due to stronger residual learning.