[Paper Review] Dueling Posterior Sampling for Preference-Based Reinforcement Learning

Ellen Novoseller, Yibing Wei+3 more|arXiv (Cornell University)|Aug 4, 2019

Advanced Bandit Algorithms Research62 references6 citations

TL;DR

This paper introduces Dueling Posterior Sampling (DPS), a Bayesian reinforcement learning algorithm that learns from trajectory-level preference feedback by combining posterior sampling with credit assignment via state-action reward models. It achieves the first theoretical regret guarantee in preference-based RL, proving asymptotic Bayesian no-regret convergence with a Bayesian linear regression credit assignment model.

ABSTRACT

In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the design of formal frameworks that admit tractable theoretical analysis remains an open challenge. Building upon ideas from preference-based bandit learning and posterior sampling in RL, we present DUELING POSTERIOR SAMPLING (DPS), which employs preference-based posterior sampling to learn both the system dynamics and the underlying utility function that governs the preference feedback. As preference feedback is provided on trajectories rather than individual state-action pairs, we develop a Bayesian approach for the credit assignment problem, translating preferences to a posterior distribution over state-action reward models. We prove an asymptotic Bayesian no-regret rate for DPS with a Bayesian linear regression credit assignment model. This is the first regret guarantee for preference-based RL to our knowledge. We also discuss possible avenues for extending the proof methodology to other credit assignment models. Finally, we evaluate the approach empirically, showing competitive performance against existing baselines.

Motivation & Objective

• To address the challenge of learning from sparse, relative preference feedback in reinforcement learning, where absolute rewards are unavailable.
• To develop a tractable Bayesian framework that jointly learns environment dynamics and the underlying reward function from pairwise trajectory preferences.
• To solve the temporal credit assignment problem by mapping trajectory-level preferences to state-action reward models.
• To provide the first theoretical regret analysis for preference-based RL, establishing asymptotic no-regret convergence.
• To empirically validate DPS across multiple environments, demonstrating robustness to credit assignment model and hyperparameter choices.

Proposed method

• DPS uses posterior sampling to generate two independent trajectories per episode, enabling preference-based learning through pairwise comparisons.
• It maintains a Bayesian posterior over state-action reward models, using preference feedback to update beliefs about which actions and states contribute to preferred outcomes.
• A credit assignment model—specifically Bayesian linear regression or Gaussian process regression—translates trajectory-level preferences into per-state-action reward estimates.
• The algorithm integrates information-theoretic techniques to bound Bayesian regret, enabling theoretical analysis of learning convergence.
• Preference feedback is modeled as a probabilistic function of the cumulative reward difference between trajectories, with noise modeled via logistic or Gaussian processes.
• Hyperparameters for credit assignment models (e.g., $σ$, $λ$, $\sigma_f^2$, $l$) are tuned via empirical validation across environments.

Experimental results

Research questions

• RQ1 Can posterior sampling be extended to handle trajectory-level preference feedback in reinforcement learning?
• RQ2 How can credit assignment be formalized to map sparse, relative preferences to per-state-action rewards?
• RQ3 What theoretical guarantees can be established for preference-based RL algorithms in the Bayesian setting?
• RQ4 How robust is the proposed method to variations in credit assignment models and hyperparameters?
• RQ5 Does the algorithm achieve competitive performance compared to existing preference-based RL baselines?

Key findings

• DPS achieves asymptotic Bayesian no-regret convergence with a Bayesian linear regression credit assignment model, marking the first regret guarantee in preference-based RL.
• Empirical evaluations in Random MDP, Mountain Car, and RiverSwim environments show DPS consistently outperforms or matches existing baselines in terms of cumulative reward.
• The algorithm demonstrates robustness to hyperparameter choices across all tested environments, with stable performance across 30–100 runs.
• Performance remains strong under varying user noise models, including logistic noise, indicating resilience to noisy preference feedback.
• The choice of credit assignment model (e.g., GP, Bayesian logistic regression) has minimal impact on final performance, suggesting generalizability.
• Theoretical analysis relies on information-theoretic bounds on posterior sampling regret, extended to the preference feedback setting.