[Paper Review] Synthetic Data-Based Simulators for Recommender Systems: A Survey

Elizaveta Stavinova, Alexander Grigorievskiy+4 more|arXiv (Cornell University)|Jun 22, 2022

Recommender Systems and Techniques4 citations

TL;DR

This survey presents a comprehensive analysis of synthetic data-based simulators for recommender systems, introducing a novel classification framework based on functionality, approbation, and industrial effectiveness. It reviews existing simulators, details their core components—synthetic data generation, scenario modeling, training/evaluation, quality control, and result summarization—and identifies key challenges and emerging trends in closing the simulation-to-reality gap.

ABSTRACT

This survey aims at providing a comprehensive overview of the recent trends in the field of modeling and simulation (M&S) of interactions between users and recommender systems and applications of the M&S to the performance improvement of industrial recommender engines. We start with the motivation behind the development of frameworks implementing the simulations -- simulators -- and the usage of them for training and testing recommender systems of different types (including Reinforcement Learning ones). Furthermore, we provide a new consistent classification of existing simulators based on their functionality, approbation, and industrial effectiveness and moreover make a summary of the simulators found in the research literature. Besides other things, we discuss the building blocks of simulators: methods for synthetic data (user, item, user-item responses) generation, methods for what-if experimental analysis, methods and datasets used for simulation quality evaluation (including the methods that monitor and/or close possible simulation-to-reality gaps), and methods for summarization of experimental simulation results. Finally, this survey considers emerging topics and open problems in the field.

Motivation & Objective

• Address the challenges of real-world training and testing of recommender systems, including high cost, data scarcity, privacy constraints, and offline-online performance inconsistency.
• Overcome limitations of historical data by leveraging synthetic data to enable controlled, scalable, and privacy-preserving simulation of user-item interactions.
• Provide a systematic classification of existing simulators based on functionality, reproducibility (approbation), and industrial applicability (effectiveness).
• Identify gaps in simulation quality evaluation, reproducibility, and benchmarking, and call for standardized methodologies to validate simulator fidelity.
• Stimulate focused research by outlining open problems such as inconsistent performance across real and synthetic data, lack of universal quality assessment, and insufficient empirical validation.

Proposed method

• Propose a new classification framework for simulators based on three criteria: functionality (presence of core components), approbation (reproducibility of experiments), and industry effectiveness (industrial deployment suitability).
• Systematically review 25+ simulators from academic and industrial sources, analyzing their goals, functional components, and real-world validation.
• Categorize and analyze five key functional components: C1 (synthetic data generation), C2 (scenario modeling), C3 (training and testing RSs), C4 (simulation quality evaluation and control), and C5 (result summarization).
• Examine methods for generating synthetic user and item profiles, user-item responses, and modeling biases (e.g., popularity, positivity) and long-term effects.
• Evaluate simulation quality through metrics such as consistency between synthetic and real-world results, bias detection, and realism of user behavior modeling.
• Highlight the importance of public code, documentation, and datasets to improve reproducibility and industrial adoption of simulators.

Experimental results

Research questions

• RQ1 How can synthetic data-based simulators effectively model complex user-item interactions while preserving realism and minimizing simulation-to-reality gaps?
• RQ2 What are the key functional components of modern simulators, and how do they contribute to training and evaluating recommender systems?
• RQ3 To what extent do existing simulators support what-if analysis, bias detection, and long-term interaction effects in recommender systems?
• RQ4 What are the main challenges in evaluating and controlling the quality of simulators, and how can these be addressed with standardized methodologies?
• RQ5 Why is there inconsistency in performance comparisons between real-world and synthetic data, and what is needed to achieve reliable benchmarking?

Key findings

• A new classification framework for simulators—based on functionality, approbation, and industry effectiveness—provides a consistent basis for comparing existing tools.
• Many simulators lack reproducibility due to limited public availability of code, datasets, and experimental configurations, hindering validation and adoption.
• Synthetic data generation methods often fail to capture real-world biases (e.g., popularity, positivity), leading to simulation-to-reality gaps that affect evaluation fidelity.
• There is no universally accepted methodology for assessing simulator quality, and current approaches remain fragmented and under-evaluated.
• Despite active research, no comprehensive experimental comparison of simulators under identical settings (datasets, metrics, RSs) has been conducted, limiting evidence-based selection.
• The practical impact of simulators—such as improvements in real-world RS performance or revenue—remains largely unquantified and lacks methodologically sound validation.