[Paper Review] A Next Basket Recommendation Reality Check

Ming Li, Sami Jullien+2 more|arXiv (Cornell University)|Sep 29, 2021

Recommender Systems and Techniques4 citations

TL;DR

This paper conducts a rigorous, fair evaluation of next basket recommendation (NBR) methods, revealing that performance gains in the literature are often misleading due to skewed focus on repetition over exploration. It proposes novel metrics to measure repetition/exploration balance and finds that repeat item prediction is significantly easier than exploration, with deep learning models underperforming on the latter, urging for task-specific model design that separates repetition and exploration strategies.

ABSTRACT

The goal of a next basket recommendation (NBR) system is to recommend items for the next basket for a user, based on the sequence of their prior baskets. Recently, a number of methods with complex modules have been proposed that claim state-of-the-art performance. They rarely look into the predicted basket and just provide intuitive reasons for the observed improvements, e.g., better representation, capturing intentions or relations, etc. We provide a novel angle on the evaluation of next basket recommendation methods, centered on the distinction between repetition and exploration: the next basket is typically composed of previously consumed items (i.e., repeat items) and new items (i.e, explore items). We propose a set of metrics that measure the repeat/explore ratio and performance of NBR models. Using these new metrics, we analyze state-of-the-art NBR models. The results of our analysis help to clarify the extent of the actual progress achieved by existing NBR methods as well as the underlying reasons for the improvements. Overall, our work sheds light on the evaluation problem of NBR and provides useful insights into the model design for this task.

Motivation & Objective

• To critically assess whether reported performance gains in NBR methods hold up under fair and comprehensive evaluation.
• To identify flaws in existing NBR evaluation practices, including inconsistent baselines, non-standard metrics, and dataset heterogeneity.
• To propose a new evaluation framework centered on the distinction between repeat (repetition) and explore (discovery) items in recommended baskets.
• To provide insights into why certain NBR models outperform others by analyzing their behavior on repetition vs. exploration tasks.
• To guide future NBR model design by highlighting the need to treat repetition and exploration as distinct subtasks with different modeling requirements.

Proposed method

• The authors conduct a large-scale comparison of three families of NBR models—frequency-based, nearest neighbor-based, and deep learning-based—on three benchmark datasets.
• They introduce new task-specific metrics to quantify the repetition/exploration ratio in recommended baskets, measuring how well models predict previously consumed items versus novel items.
• The evaluation includes user-level performance breakdowns to analyze behavior across different user groups, especially in terms of repeat vs. explore item recommendations.
• They use original implementations of published models to ensure reproducibility and avoid re-implementation bias.
• The analysis distinguishes between the difficulty of predicting repeat items (based on frequency and recency) and exploring new items (requiring modeling item correlations).
• They propose a dual-path model design strategy: using frequency/recency for repetition and neural networks for exploration, enabling controllable trade-offs between the two.

Experimental results

Research questions

• RQ1 Do the reported performance gains of state-of-the-art NBR methods hold up under a fair and comprehensive evaluation across multiple datasets and metrics?
• RQ2 To what extent are existing NBR models biased toward recommending repeat items versus exploring new items, and how does this affect long-term user engagement?
• RQ3 How do different NBR model families (frequency-based, nearest neighbor, deep learning) perform on the repeat and exploration subtasks separately?
• RQ4 Can the performance gap between repeat and explore item recommendation be explained by inherent differences in task difficulty and model capacity?
• RQ5 What design principles should guide future NBR models to better balance repetition and exploration based on user preferences?

Key findings

• No NBR method consistently outperforms all others across all datasets, indicating that method performance is highly dataset-dependent.
• All evaluated NBR methods are heavily skewed toward repetition, with significantly higher performance on repeat items than on explore items, suggesting a fundamental imbalance in task difficulty.
• Repeat item recommendation is substantially easier than exploration, with performance gaps indicating that current models are not adequately addressing the discovery of new items.
• Deep learning-based NBR models underperform on the exploration task compared to simpler baselines, challenging the assumption that complex models are always better for NBR.
• The proposed repetition/exploration metrics reveal that models often fail to recommend novel items even when users have a history of exploring, highlighting a critical gap in current evaluation and design.
• Separating repetition and exploration in model design—using frequency/recency for repetition and neural networks for exploration—can lead to more effective and controllable NBR systems.