[Paper Review] Deep Embedding Forest: Forest-based Serving with Deep Embedding Features

Jie Zhu, Ying Shan+4 more|arXiv (Cornell University)|Mar 15, 2017

Face and Expression Recognition18 references4 citations

TL;DR

This paper proposes Deep Embedding Forest, a hybrid model that combines deep neural network (DNN) embedding layers with tree-based models (e.g., XGBoost, LightGBM) to achieve DNN-level performance with forest-level inference speed. By training embedding layers to map high-dimensional features into low-dimensional vectors and using a tree ensemble for fast inference, the method reduces serving time to a fraction of DNNs while maintaining near-optimal accuracy on large-scale sponsored search data.

ABSTRACT

Deep Neural Networks (DNN) have demonstrated superior ability to extract high level embedding vectors from low level features. Despite the success, the serving time is still the bottleneck due to expensive run-time computation of multiple layers of dense matrices. GPGPU, FPGA, or ASIC-based serving systems require additional hardware that are not in the mainstream design of most commercial applications. In contrast, tree or forest-based models are widely adopted because of low serving cost, but heavily depend on carefully engineered features. This work proposes a Deep Embedding Forest model that benefits from the best of both worlds. The model consists of a number of embedding layers and a forest/tree layer. The former maps high dimensional (hundreds of thousands to millions) and heterogeneous low-level features to the lower dimensional (thousands) vectors, and the latter ensures fast serving. Built on top of a representative DNN model called Deep Crossing, and two forest/tree-based models including XGBoost and LightGBM, a two-step Deep Embedding Forest algorithm is demonstrated to achieve on-par or slightly better performance as compared with the DNN counterpart, with only a fraction of serving time on conventional hardware. After comparing with a joint optimization algorithm called partial fuzzification, also proposed in this paper, it is concluded that the two-step Deep Embedding Forest has achieved near optimal performance. Experiments based on large scale data sets (up to 1 billion samples) from a major sponsored search engine proves the efficacy of the proposed model.

Motivation & Objective

• To address the high inference cost of deep neural networks (DNNs) in production systems despite their strong performance.
• To overcome the limitations of tree-based models that require extensive manual feature engineering for high-dimensional, heterogeneous features.
• To develop a hybrid architecture that leverages DNNs for automatic feature learning and tree models for low-cost inference.
• To enable efficient, scalable serving on standard CPU hardware without specialized accelerators (e.g., GPUs, FPGAs).
• To demonstrate that a two-step training strategy can achieve near-optimal performance compared to joint optimization.

Proposed method

• The model uses a DNN backbone (based on Deep Crossing) to learn low-dimensional dense embeddings from high-dimensional, heterogeneous features such as text and categorical variables.
• The output of the embedding layers is fed into a tree-based model (e.g., XGBoost or LightGBM), which performs fast inference via decision paths.
• A two-step training process first initializes the embedding layers using pre-trained DNN weights, then trains the tree layer on the embedded features.
• A joint optimization technique called partial fuzzification is introduced to refine the tree parameters and embedding representations simultaneously through gradient backpropagation.
• The tree layer uses a differentiable soft-split mechanism where each internal node applies a sigmoid-based routing function to determine the probability of going left or right, enabling gradient flow through the tree structure.
• The final prediction is computed as a weighted sum of leaf node outputs, with the raw score transformed via a sigmoid function to produce the final probability.

Experimental results

Research questions

• RQ1 Can a hybrid model combining DNNs and tree-based models achieve DNN-level performance while maintaining the low inference cost of tree models?
• RQ2 How effective is a two-step training strategy compared to end-to-end joint optimization in this hybrid setting?
• RQ3 To what extent can DNN-learned embeddings improve the performance of tree-based models without requiring manual feature engineering?
• RQ4 Can the proposed model scale to billion-sample datasets and maintain low inference latency on standard CPU hardware?
• RQ5 How does the partial fuzzification algorithm compare to alternative joint optimization methods in terms of convergence and final performance?

Key findings

• The two-step Deep Embedding Forest method achieves on-par or slightly better performance than the full DNN baseline on large-scale sponsored search data.
• Serving time is reduced to a fraction of that of the DNN model, achieving near-tree-level inference speed on standard CPU hardware.
• The model demonstrates strong scalability, successfully training and serving on datasets up to 1 billion samples.
• The partial fuzzification joint optimization method converges to near-optimal performance, validating the effectiveness of the two-step approach.
• The model maintains high performance across different tree-based backbones, including XGBoost and LightGBM, showing robustness to model choice.
• The use of differentiable soft-splits enables effective backpropagation through the tree structure, allowing joint optimization of embeddings and tree parameters.