[Paper Review] DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving

Wei Deng, Junwei Pan+4 more|arXiv (Cornell University)|Feb 17, 2020

Recommender Systems and Techniques59 references4 citations

TL;DR

DeepLight proposes a lightweight, pruned neural network framework to accelerate click-through rate (CTR) prediction in real-time ad serving. By jointly optimizing structural pruning of deep neural network layers, sparse embedding vectors, and a field-weighted matrix for feature interactions, it achieves 46X speedup on Criteo and 27X on Avazu with no accuracy loss, enabling efficient deployment of complex models in production systems.

ABSTRACT

Click-through rate (CTR) prediction is a crucial task in online display advertising. The embedding-based neural networks have been proposed to learn both explicit feature interactions through a shallow component and deep feature interactions using a deep neural network (DNN) component. These sophisticated models, however, slow down the prediction inference by at least hundreds of times. To address the issue of significantly increased serving delay and high memory usage for ad serving in production, this paper presents \emph{DeepLight}: a framework to accelerate the CTR predictions in three aspects: 1) accelerate the model inference via explicitly searching informative feature interactions in the shallow component; 2) prune redundant layers and parameters at intra-layer and inter-layer level in the DNN component; 3) promote the sparsity of the embedding layer to preserve the most discriminant signals. By combining the above efforts, the proposed approach accelerates the model inference by 46X on Criteo dataset and 27X on Avazu dataset without any loss on the prediction accuracy. This paves the way for successfully deploying complicated embedding-based neural networks in production for ad serving.

Motivation & Objective

• Address the high inference latency and memory overhead of deep embedding-based CTR models in real-time ad serving systems.
• Overcome the trade-off between model accuracy and inference efficiency in production environments.
• Develop a framework that enables fast, low-memory inference while preserving state-of-the-art prediction performance.
• Enable practical deployment of complex deep learning models in large-scale online advertising systems with strict latency and resource constraints.

Proposed method

• Introduce DeepFwFM, a field-weighted embedding-based neural network with a sparse-structured field matrix for efficient feature interaction modeling.
• Apply structural pruning at the inter-layer level in the deep neural network (DNN) component to remove redundant parameters while maintaining performance.
• Prune dense embedding vectors to sparse representations, significantly reducing memory usage and improving inference speed.
• Use a preconditioner-based optimization strategy to guide pruning while preserving model accuracy during compression.
• Combine multiple pruning strategies—DNN layer pruning, field matrix sparsity, and embedding vector sparsification—into a unified framework called DeepLight.
• Leverage the inherent sparsity of the field-weighted matrix structure to enable high compression rates without performance degradation.

Experimental results

Research questions

• RQ1 Can structural pruning of deep neural network layers and embedding vectors accelerate CTR prediction without sacrificing accuracy?
• RQ2 How effective is the field-weighted matrix in enabling both high performance and high sparsity for efficient inference?
• RQ3 What is the optimal trade-off between model compression (latency, memory) and prediction accuracy in real-world ad serving systems?
• RQ4 Can a unified pruning strategy across DNN, field matrix, and embedding components achieve superior speedup compared to pruning individual components in isolation?

Key findings

• DeepLight achieves a 46X speedup on the Criteo dataset and 27X on the Avazu dataset with no loss in AUC, demonstrating significant inference acceleration.
• On Criteo, the model maintains an AUC of 0.8114 with 99% pruning in the DNN component, 95% in the field matrix, and 40% in embeddings, achieving 0.093ms inference latency.
• On Avazu, the model achieves 27X speedup with 98% DNN pruning, 90% field matrix pruning, and 0% embedding pruning, maintaining an AUC of 0.7894.
• The framework reduces memory usage by up to 10X on Criteo and 2.5X on Avazu, enabling efficient model storage and retrieval in online serving.
• DeepLight outperforms sparse versions of DeepFM, NFM, and xDeepFM in both speed and accuracy, achieving 0.8114 AUC on Criteo compared to 0.8032 for sparse DeepFM.
• The field-weighted matrix structure enables high sparsity and efficient pruning, making it uniquely suited for structural compression in CTR models.