[Paper Review] Churn Prediction with Sequential Data and Deep Neural Networks. A Comparative Analysis

C. Gary Mena, Arno De Caigny+3 more|arXiv (Cornell University)|Sep 24, 2019

Customer churn and segmentationBusiness, Management and Accounting16 references22 citations

TL;DR

This paper proposes using Long Short-Term Memory (LSTM) networks to model sequential Recency, Frequency, and Monetary (RFM) data for customer churn prediction in financial services. Results show that LSTMs outperform regularized logistic regression with static features, achieving a top-decile lift of 4.211 and a 25% improvement in out-of-sample performance when LSTM-predicted probabilities are used as features in logistic models.

ABSTRACT

Off-the-shelf machine learning algorithms for prediction such as regularized logistic regression cannot exploit the information of time-varying features without previously using an aggregation procedure of such sequential data. However, recurrent neural networks provide an alternative approach by which time-varying features can be readily used for modeling. This paper assesses the performance of neural networks for churn modeling using recency, frequency, and monetary value data from a financial services provider. Results show that RFM variables in combination with LSTM neural networks have larger top-decile lift and expected maximum profit metrics than regularized logistic regression models with commonly-used demographic variables. Moreover, we show that using the fitted probabilities from the LSTM as feature in the logistic regression increases the out-of-sample performance of the latter by 25 percent compared to a model with only static features.

Motivation & Objective

• To evaluate the predictive performance of LSTM neural networks using sequential RFM data for churn modeling in financial services.
• To compare LSTM-based models with traditional off-the-shelf machine learning models that rely on static features.
• To investigate whether LSTM-fitted probabilities can effectively summarize sequential RFM data for use in conventional models.
• To assess the impact of different aggregation techniques for time-varying features on model performance.
• To determine the optimal integration strategy for sequential behavioral data in churn prediction frameworks.

Proposed method

• Employed Long Short-Term Memory (LSTM) recurrent neural networks to model sequential RFM variables directly without aggregation.
• Used l1-regularized logistic regression as a baseline model with static customer demographics, behavior, and contact variables.
• Generated out-of-sample predicted probabilities from the LSTM model and used them as additional features in logistic regression.
• Evaluated model performance using cross-validation with three outer folds to ensure robustness.
• Compared multiple feature engineering strategies for RFM data, including aggregation, normalization, and lagged values.
• Applied standard evaluation metrics: AUC, top-decile lift, and Expected Maximum Profit Measure (EMPC) for model comparison.

Experimental results

Research questions

• RQ1 How does the predictive performance of LSTM models using raw sequential RFM data compare to regularized logistic regression with static features?
• RQ2 Can the fitted probabilities from an LSTM model effectively summarize sequential RFM information for use in traditional machine learning models?
• RQ3 Does incorporating LSTM-derived probabilities into logistic regression improve out-of-sample performance compared to using aggregated RFM features?
• RQ4 Which combination of static features and sequential data representations yields the highest top-decile lift and EMPC?
• RQ5 Is there a significant performance gain when using deep learning to model sequential behavior versus traditional aggregation methods?

Key findings

• The LSTM model achieved a top-decile lift of 4.211, significantly outperforming the logistic regression model with only static features, which had a lift of 3.350.
• Using LSTM-fitted probabilities as features in logistic regression increased the out-of-sample performance by 25.7% compared to a model with only static features.
• The logistic model augmented with LSTM probabilities achieved an Expected Maximum Profit Measure (EMPC) of 0.012, which was three times higher than the baseline model with only static features.
• Adding aggregated or normalized lagged RFM features to the logistic model with LSTM probabilities did not improve performance beyond the model using only LSTM probabilities, indicating that LSTM probabilities effectively summarize sequential information.
• The LSTM model itself achieved an AUC of 0.741 and a top-decile lift of 4.101, demonstrating strong performance despite not being the best-performing model in all metrics.
• Lift curves from cross-validation folds confirmed that the LSTM model and the logistic model with LSTM probabilities consistently outperformed other configurations in the top decile.