[Paper Review] Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret

Ming Shi, Yingbin Liang+2 more|arXiv (Cornell University)|Mar 20, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

The paper develops RL from multi-source imperfect preferences (RL-MSIP) and proves a regret bound that interpolates between M-dependent statistical gains and robustness to cumulative imperfection budget, with a matching lower bound and a counterexample to naive aggregation.

ABSTRACT

Reinforcement learning from human feedback (RLHF) replaces hard-to-specify rewards with pairwise trajectory preferences, yet regret-oriented theory often assumes that preference labels are generated consistently from a single ground-truth objective. In practical RLHF systems, however, feedback is typically \emph{multi-source} (annotators, experts, reward models, heuristics) and can exhibit systematic, persistent mismatches due to subjectivity, expertise variation, and annotation/modeling artifacts. We study episodic RL from \emph{multi-source imperfect preferences} through a cumulative imperfection budget: for each source, the total deviation of its preference probabilities from an ideal oracle is at most $ω$ over $K$ episodes. We propose a unified algorithm with regret $ ilde{O}(\sqrt{K/M}+ω)$, which exhibits a best-of-both-regimes behavior: it achieves $M$-dependent statistical gains when imperfection is small (where $M$ is the number of sources), while remaining robust with unavoidable additive dependence on $ω$ when imperfection is large. We complement this with a lower bound $ ildeΩ(\max\{\sqrt{K/M},ω\})$, which captures the best possible improvement with respect to $M$ and the unavoidable dependence on $ω$, and a counterexample showing that naïvely treating imperfect feedback as as oracle-consistent can incur regret as large as $ ildeΩ(\min\{ω\sqrt{K},K\})$. Technically, our approach involves imperfection-adaptive weighted comparison learning, value-targeted transition estimation to control hidden feedback-induced distribution shift, and sub-importance sampling to keep the weighted objectives analyzable, yielding regret guarantees that quantify when multi-source feedback provably improves RLHF and how cumulative imperfection fundamentally limits it.

Motivation & Objective

• Motivate RL with multi-source imperfect trajectory preferences in RLHF settings.
• Quantify how the number of sources M and cumulative imperfection budget ω affect regret.
• Develop an algorithm RL-MSIP that adapts to imperfection levels and achieves favorable regret.
• Provide lower bounds and a counterexample to naive aggregation to tighten understanding of imperfection effects.

Proposed method

• Formulate multi-source imperfect preference feedback with a cumulative budget ω across K episodes.
• Propose imperfection-adaptive weighted comparison learning to estimate the comparison function.
• Use value-targeted transition estimation to control distribution shift from feedback.
• Implement policy-level optimism with Bounded UCB to balance exploration under preference-only feedback.
• Apply sub-importance sampling to keep weighted objectives analyzable and stable.

Experimental results

Research questions

• RQ1 How does the number of sources M and cumulative imperfection ω influence regret in RLHF with imperfect preferences?
• RQ2 Can we design an algorithm that achieves both M-dependent gains when imperfection is small and robustness when imperfection is large?
• RQ3 What are the fundamental limits (lower bounds) for regret under multi-source imperfect preferences?
• RQ4 What pitfalls arise from naïvely aggregating imperfect preferences, and can we quantify their impact?
• RQ5 How should we estimate transitions and preferences to maintain tractable regret analysis under imperfection?

Key findings

• RL-MSIP achieves a regret of roughly Õ(√(K/M) + ω).
• A lower bound shows regret must be at least Õ(max{√(K/M), ω}).
• There exists a counterexample where ignoring imperfection yields Õ(min{ω√K, K}).
• The approach combines imperfection-adaptive weighting, value-targeted regression, policy-level optimism, and sub-importance sampling.
• The results quantify when multi-source feedback improves RLHF and how imperfection limits it.