[Paper Review] Enhancing Bandit Algorithms with LLMs for Time-varying User Preferences in Streaming Recommendations

Chenglei Shen, Yi Zhan+3 more|arXiv (Cornell University)|Feb 8, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

HyperBandit+ introduces a time-aware hypernetwork and LLM-based warm-start to handle time-varying user preferences in streaming recommendations, with sublinear regret guarantees and strong empirical performance.

ABSTRACT

In real-world streaming recommender systems, user preferences evolve dynamically over time. Existing bandit-based methods treat time merely as a timestamp, neglecting its explicit relationship with user preferences and leading to suboptimal performance. Moreover, online learning methods often suffer from inefficient exploration-exploitation during the early online phase. To address these issues, we propose HyperBandit+, a novel contextual bandit policy that integrates a time-aware hypernetwork to adapt to time-varying user preferences and employs a large language model-assisted warm-start mechanism (LLM Start) to enhance exploration-exploitation efficiency in the early online phase. Specifically, HyperBandit+ leverages a neural network that takes time features as input and generates parameters for estimating time-varying rewards by capturing the correlation between time and user preferences. Additionally, the LLM Start mechanism employs multi-step data augmentation to simulate realistic interaction data for effective offline learning, providing warm-start parameters for the bandit policy in the early online phase. To meet real-time streaming recommendation demands, we adopt low-rank factorization to reduce hypernetwork training complexity. Theoretically, we rigorously establish a sublinear regret upper bound that accounts for both the hypernetwork and the LLM warm-start mechanism. Extensive experiments on real-world datasets demonstrate that HyperBandit+ consistently outperforms state-of-the-art baselines in terms of accumulated rewards.

Motivation & Objective

• Motivate handling time-varying user preferences in streaming recommendations.
• Model time-period dependent rewards via a time-varying user preference matrix.
• Enhance exploration–exploitation in cold-start via LLM-based offline warm start.
• Reduce training complexity with low-rank hypernetwork parameterization.
• Provide theoretical regret guarantees for the proposed framework.

Proposed method

• Introduce HyperBandit+ which uses a time-period aware hypernetwork to generate user preference mappings.
• Define the true time-varying reward as r*(u,a,p) = c_a^T Theta_p^* c_u and learn Theta_p over time periods.
• Employ LLM Start and LLM-Enhanced Embedding to augment latent and observed features respectively.
• Use low-rank factorization to compress hypernetwork outputs and accelerate online training.
• Offline phase includes Euler Embedding and LLM-Enhanced Embedding for side information; LLM Start simulates data for warm start.
• Update the policy online via ridge regression with a closed-form solution for latent features.

Experimental results

Research questions

• RQ1 How to explicitly model and adapt to periodic/time-varying user preferences in a streaming contextual bandit?
• RQ2 Can a hypernetwork generate time-period specific user preference matrices to improve recommendation accuracy?
• RQ3 Does LLM-based data augmentation provide a meaningful warm start to mitigate early-stage exploration–exploitation issues?
• RQ4 What are the theoretical regret guarantees for a bandit policy that incorporates time-aware hypernetworks and LLM-based warm starts?

Key findings

• HyperBandit+ achieves sublinear regret that accounts for the hypernetwork and LLM warm-start mechanisms.
• Empirical results on real-world datasets show HyperBandit+ consistently outperforms state-of-the-art baselines in accumulated rewards.
• Low-rank factorization reduces training complexity without sacrificing performance.
• LLM Start provides effective offline data augmentation, improving early online exploration–exploitation efficiency.
• LLM-Enhanced Embedding improves side information quality by enriching user and item representations.