[Paper Review] Preference Transformer: Modeling Human Preferences using Transformers for RL

Chang‐Yeon Kim, Jongjin Park+4 more|arXiv (Cornell University)|Mar 2, 2023

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes Preference Transformer, a transformer-based architecture that models human preferences using a weighted sum of non-Markovian rewards, enabling more accurate and temporally aware reward inference. It outperforms prior methods in learning complex control behaviors—such as double backflips in Hopper—using real human feedback, while automatically identifying critical events in trajectories through self-attention mechanisms.

ABSTRACT

Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.

Motivation & Objective

• To address the limitations of Markovian reward assumptions in preference-based RL, which fail to capture temporal dependencies in human decision-making.
• To improve scalability and performance of preference-based RL by modeling human preferences through a weighted sum of non-Markovian rewards.
• To design a neural architecture that can infer critical events in trajectories and assign appropriate importance weights using self-attention.
• To enable sample-efficient learning of complex behaviors—such as multi-backflips and robotic manipulation—using real human feedback.

Proposed method

• Proposes a novel preference model based on the weighted sum of non-Markovian rewards, where each reward is dynamically weighted by its importance in the trajectory.
• Introduces a transformer-based architecture with stacked causal and bidirectional self-attention layers to generate non-Markovian rewards and their importance weights.
• Designs a novel 'preference attention' layer that computes attention over trajectory segments to infer critical events and assign dynamic weights.
• Uses the weighted sum of non-Markovian rewards as the preference predictor, which is trained to align with human pairwise comparisons.
• Employs a two-stream architecture: one for processing each trajectory segment and another for computing the preference score via the weighted sum.
• Leverages the transformer's ability to model long-range dependencies, enabling better credit assignment across complex, temporally extended behaviors.

Experimental results

Research questions

• RQ1 Can a non-Markovian reward modeling approach improve preference-based RL performance compared to Markovian assumptions?
• RQ2 Can a transformer-based architecture effectively learn importance weights for different events in a trajectory to reflect human decision-making?
• RQ3 Does the proposed method enable agents to learn complex, long-horizon behaviors like double backflips using only human preference feedback?
• RQ4 Can the learned attention weights in the model identify and highlight critical events that influence human preference judgments?
• RQ5 How does the model compare to prior preference modeling methods in terms of sample efficiency and generalization across diverse control tasks?

Key findings

• Preference Transformer successfully learned to perform double backflips in the Hopper environment using only 300 human feedback queries, while a Markovian reward function failed to learn the behavior.
• The model achieved state-of-the-art performance on D4RL navigation and locomotion benchmarks, as well as Robomimic robotic manipulation tasks, using real human preferences.
• The learned importance weights in the preference attention layer clearly highlighted critical events such as takeoff and landing phases in the backflip trajectory, aligning with human attention patterns.
• The model demonstrated robustness and generalization across diverse control tasks, including those with non-Markovian dynamics and multi-objective objectives.
• Visual analysis of attention maps confirmed that the model captures temporal dependencies and assigns higher weights to events that significantly influence human preference.