[Paper Review] Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems

Zhe Chen, Yuyan Wang+5 more|arXiv (Cornell University)|Aug 17, 2020

Stochastic Gradient Optimization Techniques49 references4 citations

TL;DR

This paper proposes a novel method to infer ensemble prediction variation in recommender systems using neuron activation strength, avoiding the need for expensive multiple inference runs. It achieves high R² scores (0.56 on MovieLens, 0.81 on Criteo) and strong AUC performance (0.92 for low, 0.89 for high variation buckets), demonstrating that activation patterns alone can effectively estimate uncertainty without ensemble inference.

ABSTRACT

Despite deep neural network (DNN)'s impressive prediction performance in various domains, it is well known now that a set of DNN models trained with the same model specification and the same data can produce very different prediction results. Ensemble method is one state-of-the-art benchmark for prediction uncertainty estimation. However, ensembles are expensive to train and serve for web-scale traffic. In this paper, we seek to advance the understanding of prediction variation estimated by the ensemble method. Through empirical experiments on two widely used benchmark datasets MovieLens and Criteo in recommender systems, we observe that prediction variations come from various randomness sources, including training data shuffling, and parameter random initialization. By introducing more randomness into model training, we notice that ensemble's mean predictions tend to be more accurate while the prediction variations tend to be higher. Moreover, we propose to infer prediction variation from neuron activation strength and demonstrate the strong prediction power from activation strength features. Our experiment results show that the average R squared on MovieLens is as high as 0.56 and on Criteo is 0.81. Our method performs especially well when detecting the lowest and highest variation buckets, with 0.92 AUC and 0.89 AUC respectively. Our approach provides a simple way for prediction variation estimation, which opens up new opportunities for future work in many interesting areas (e.g.,model-based reinforcement learning) without relying on serving expensive ensemble models.

Motivation & Objective

• To understand the sources and nature of prediction variation in ensemble deep neural networks used in recommender systems.
• To investigate whether neuron activation strength can serve as a proxy for prediction variation, reducing reliance on computationally expensive ensembles.
• To develop and validate a lightweight method for estimating prediction uncertainty using only activation features from a single forward pass.
• To enable scalable uncertainty estimation in real-time, web-scale recommendation systems by replacing full ensembles with activation-based inference.

Proposed method

• The authors train multiple DNN models with controlled randomness (e.g., data shuffling, weight initialization) to generate ensemble predictions and quantify prediction variation per input.
• Prediction variation is defined as the variance of predictions across ensemble members for each input example.
• Neuron activation strength is extracted from a single forward pass of a DNN, using activation values and binary on/off states of neurons.
• A supervised regression model is trained to predict ensemble prediction variation using activation strength features as input.
• The method is evaluated on MovieLens (regression) and Criteo (binary classification) datasets, with performance measured via R² and AUC.
• The authors use a ground-truth ensemble of 1,000 models to compute reference prediction variation, and assess the performance of smaller ensembles via delta ratio to determine optimal ensemble size.

Experimental results

Research questions

• RQ1 What are the primary sources of prediction variation in DNN ensembles trained with identical specifications and data?
• RQ2 How does increasing randomness in training (e.g., data shuffling, weight initialization) affect the accuracy and variation of ensemble predictions?
• RQ3 Can neuron activation strength from a single forward pass reliably predict the variation observed across a full ensemble?
• RQ4 How well does the activation-based method perform in detecting extreme variation cases (e.g., highest and lowest variation buckets)?

Key findings

• Prediction variation in DNN ensembles arises from multiple randomness sources, including data shuffling and parameter initialization, and increases with more randomness.
• Ensembles trained with higher randomness produce more accurate mean predictions but also higher prediction variation, indicating a trade-off between accuracy and uncertainty.
• Neuron activation strength features achieve an R² of 0.56 on MovieLens and 0.81 on Criteo in predicting ensemble prediction variation, demonstrating strong predictive power.
• The method performs exceptionally well in detecting extreme variation buckets, with AUC of 0.92 for the lowest variation and 0.89 for the highest variation.
• The delta ratio analysis shows that an ensemble size of 100 captures approximately 93% of the prediction variation from a ground-truth ensemble of 1,000 models.