[Paper Review] Explore-Exploit: A Framework for Interactive and Online Learning

Honglei Liu, Anuj Kumar+2 more|arXiv (Cornell University)|Dec 1, 2018

Advanced Bandit Algorithms ResearchDecision Sciences13 references3 citations

TL;DR

Explore-Exploit is a production-ready framework for online and interactive machine learning that balances exploration and exploitation to minimize user experience regressions. It uses configurable online learning operators—such as UCB1Enhanced and epsilon-greedy—across tasks like hyperparameter tuning, active learning, and reinforcement learning, achieving automatic convergence to optimal settings in days, not weeks.

ABSTRACT

Interactive user interfaces need to continuously evolve based on the interactions that a user has (or does not have) with the system. This may require constant exploration of various options that the system may have for the user and obtaining signals of user preferences on those. However, such an exploration, especially when the set of available options itself can change frequently, can lead to sub-optimal user experiences. We present Explore-Exploit: a framework designed to collect and utilize user feedback in an interactive and online setting that minimizes regressions in end-user experience. This framework provides a suite of online learning operators for various tasks such as personalization ranking, candidate selection and active learning. We demonstrate how to integrate this framework with run-time services to leverage online and interactive machine learning out-of-the-box. We also present results demonstrating the efficiencies that can be achieved using the Explore-Exploit framework.

Motivation & Objective

• Address the challenge of continuously adapting systems to evolving user preferences in real-time production environments.
• Minimize user experience regressions caused by suboptimal exploration during online learning.
• Enable seamless integration of interactive and online learning into existing run-time services without architectural overhaul.
• Support diverse learning tasks such as ranking, candidate selection, hyperparameter tuning, and active learning with configurable, plug-and-play operators.
• Provide a production-grade system that automates feedback-driven model adaptation while maintaining high relevance and usability.

Proposed method

• Design a subscription-based service architecture allowing any run-time system to opt in or out of exploration with minimal coupling.
• Define an Exploration Target with Transformers, Operators, Feedback Fetchers, and metadata to encapsulate learning objectives.
• Implement online learning operators including epsilon-greedy, UCB1, Thompson Sampling, and a novel UCB1Enhanced variant for target-specific optimization.
• Use the UCB1Enhanced operator to balance exploration and exploitation via a reward-based objective function that penalizes deviation from a target metric (e.g., CTR = 0.11).
• Log and process real-time user feedback signals to dynamically adjust candidate selection and model parameters.
• Support both streaming and pool-based active learning, and integrate with reinforcement learning and AutoML workflows.

Experimental results

Research questions

• RQ1 How can exploration and exploitation be effectively balanced in real-time production systems to avoid user experience degradation?
• RQ2 What architectural patterns enable seamless integration of online learning into existing run-time services without requiring system-wide changes?
• RQ3 Can automated hyperparameter tuning via online learning converge to optimal settings faster and more reliably than manual tuning?
• RQ4 How effective is the UCB1Enhanced operator in minimizing regret while achieving target performance metrics (e.g., CTR) in online learning settings?
• RQ5 To what extent can a unified framework support diverse learning tasks such as ranking, active learning, and reinforcement learning in production?

Key findings

• The Explore-Exploit framework enabled automatic convergence to an optimal threshold (0.12) for a personalization model with a target CTR of 0.11, reducing manual tuning from weeks to days.
• After a few days of operation, the system converged to the candidate with the smallest relative deviation from the target reward (0.12, deviation ≈ 0), confirming the accuracy of the learning process.
• Initial exploration was uniform across candidates, demonstrating unbiased data collection before convergence, which supports the validity of the learning process.
• The framework successfully reduced engineering effort by eliminating the need for manual A/B testing and iterative threshold tuning across multiple models.
• The UCB1Enhanced operator effectively prioritized high-potential candidates, minimizing regret while maintaining exploration of under-explored options.
• The subscription-based design allowed for easy onboarding and removal of online learning functionality, enabling flexible deployment across different services.