[Paper Review] Predicting purchasing intent: Automatic Feature Learning using Recurrent Neural Networks

Humphrey Sheil, Omer Rana+1 more|arXiv (Cornell University)|Jul 21, 2018

Consumer Market Behavior and Pricing28 references18 citations

TL;DR

This paper proposes a deep learning approach using Long Short-Term Memory (LSTM) networks with learnable embeddings to predict user purchasing intent in E-commerce without manual feature engineering. The model achieves 98.4% of state-of-the-art performance on one benchmark and exceeds it on another, demonstrating that automatic feature learning via RNNs can match or surpass traditional methods like Gradient Boosting Machines while reducing dependency on domain-specific feature engineering.

ABSTRACT

We present a neural network for predicting purchasing intent in an Ecommerce setting. Our main contribution is to address the significant investment in feature engineering that is usually associated with state-of-the-art methods such as Gradient Boosted Machines. We use trainable vector spaces to model varied, semi-structured input data comprising categoricals, quantities and unique instances. Multi-layer recurrent neural networks capture both session-local and dataset-global event dependencies and relationships for user sessions of any length. An exploration of model design decisions including parameter sharing and skip connections further increase model accuracy. Results on benchmark datasets deliver classification accuracy within 98% of state-of-the-art on one and exceed state-of-the-art on the second without the need for any domain / dataset-specific feature engineering on both short and long event sequences.

Motivation & Objective

• To reduce reliance on extensive, domain-specific feature engineering in E-commerce user intent prediction.
• To develop a deep learning model that automatically learns representations from raw clickstream data.
• To evaluate whether RNNs with learnable embeddings can match or exceed performance of state-of-the-art models like Gradient Boosted Machines.
• To assess model generalization across different E-commerce datasets with varying session length distributions.
• To demonstrate that a minimal, hardware-efficient deep learning model can achieve competitive performance with standard industry methods.

Proposed method

• Uses multi-layer Long Short-Term Memory (LSTM) networks to model sequential user interactions in E-commerce sessions.
• Employs trainable embedding layers to represent categorical items, quantities, and unique instances in a shared vector space.
• Applies parameter sharing across time steps to improve generalization and reduce overfitting on short sequences.
• Uses skip connections to stabilize training and improve gradient flow through deep architectures.
• Leverages end-to-end backpropagation to allow gradients to update both embeddings and network weights simultaneously.
• Trains the model using cross-entropy loss with early stopping and Adam optimization on a single GPU.

Experimental results

Research questions

• RQ1 Can a recurrent neural network with learnable embeddings achieve state-of-the-art performance in user purchase intent prediction without manual feature engineering?
• RQ2 How does the performance of the proposed RNN model compare to Gradient Boosted Machines on benchmark E-commerce datasets?
• RQ3 What impact do architectural choices like skip connections and parameter sharing have on model accuracy and training stability?
• RQ4 How well does the model generalize to datasets with different session length distributions, especially those dominated by single-click sessions?
• RQ5 Can the model maintain high performance with minimal hardware requirements, making it accessible for real-world deployment?

Key findings

• The proposed LSTM model achieves 98.4% of the state-of-the-art AUC performance on the first benchmark dataset without any feature engineering.
• On the second dataset, the model exceeds state-of-the-art performance, achieving an AUC of 0.837 compared to the GBM baseline of 0.834.
• Freezing the embedding layer reduced model performance to an AUC of 0.808 and increased training time by 3x, demonstrating the importance of end-to-end training.
• The model outperforms GBM on short sessions (especially single-click sessions), where GBM lacks features like dwell time or sequence context.
• The model trains in approximately 6 hours on a single GPU with 1–2 GB RAM, significantly reducing resource demands compared to prior work using 150 machines.
• The model generalizes well across datasets, maintaining strong performance even when session length distributions differ significantly.