[Paper Review] Learning a Universal Human Prior for Dexterous Manipulation from Human Preference

Zihan Ding, Yuanpei Chen+5 more|arXiv (Cornell University)|Apr 10, 2023

Robot Manipulation and Learning4 citations

TL;DR

This paper proposes a reinforcement learning from human feedback (RLHF) framework that learns a universal human prior for dexterous manipulation using human preference judgments on robot trajectory videos. By iteratively generating diverse policies, collecting human preferences, and training a task-agnostic reward model, the method improves policy naturalness and task performance across 20 simulated dexterous manipulation tasks—achieving a 22.3% improvement in preference probability over original policies after four fine-tuning iterations, with generalization to unseen tasks and sim-to-real transfer.

ABSTRACT

Generating human-like behavior on robots is a great challenge especially in dexterous manipulation tasks with robotic hands. Scripting policies from scratch is intractable due to the high-dimensional control space, and training policies with reinforcement learning (RL) and manual reward engineering can also be hard and lead to unnatural motions. Leveraging the recent progress on RL from Human Feedback, we propose a framework that learns a universal human prior using direct human preference feedback over videos, for efficiently tuning the RL policies on 20 dual-hand robot manipulation tasks in simulation, without a single human demonstration. A task-agnostic reward model is trained through iteratively generating diverse polices and collecting human preference over the trajectories; it is then applied for regularizing the behavior of polices in the fine-tuning stage. Our method empirically demonstrates more human-like behaviors on robot hands in diverse tasks including even unseen tasks, indicating its generalization capability.

Motivation & Objective

• To address the challenge of training dexterous manipulation policies that exhibit human-like, natural motion, avoiding unnatural or jarring behaviors common in standard RL training.
• To reduce reliance on costly human demonstrations or manual reward engineering by leveraging human preference feedback as a scalable proxy for human behavioral norms.
• To learn a single, task-agnostic reward model that generalizes across diverse dexterous manipulation tasks, including unseen ones.
• To enable efficient policy fine-tuning in simulation and sim-to-real transfer using only preference signals, without demonstrations.
• To investigate whether human preference feedback can effectively regularize policies toward energy-efficient, joint-safe, and human-norm-compliant behaviors.

Proposed method

• An iterative pipeline alternates between generating diverse policies across 20 dexterous manipulation tasks, collecting human preference labels on video trajectories, and training a task-agnostic reward model (RM) from the labeled feedback.
• The reward model is trained using a preference learning objective that compares pairs of trajectories, learning to predict which motion appears more human-like.
• Policies are fine-tuned using a hybrid objective that combines task-specific reward (e.g., success signal) and the human preference reward from the RM, with adaptive scaling to balance both signals.
• The method uses PPO with an entropy bonus to encourage exploration during policy fine-tuning, while the RM guides behavior toward human norms.
• A dedicated platform was built to collect human preference feedback on robot motion videos, ensuring diverse and high-quality labeling across tasks.
• The final RM is applied to fine-tune policies in both seen and unseen tasks, enabling zero-shot generalization and sim-to-real transfer.

Experimental results

Research questions

• RQ1 Can human preference feedback on video trajectories effectively learn a universal human prior for dexterous manipulation without demonstrations?
• RQ2 How well can a single, task-agnostic reward model generalize to unseen dexterous manipulation tasks?
• RQ3 Does fine-tuning policies with a human preference reward model improve both task performance and human-likeness of motion?
• RQ4 What hyperparameter choices (e.g., scaling coefficients) significantly affect the balance between task success and human-likeness in policy fine-tuning?
• RQ5 In which tasks does the RM fail to guide policies toward human-like behavior, and why?

Key findings

• The method achieved a 22.3% improvement in preference probability over original RL policies after four iterations of fine-tuning, demonstrating strong effectiveness in learning human-like behavior.
• The task-agnostic reward model successfully generalized to unseen dexterous manipulation tasks, improving both preference scores and task success rates.
• Sim-to-real transfer was successfully achieved, indicating the method’s robustness and practical deployability on real robotic systems.
• Using the hybrid objective (Eq. 3) that balances task reward and human preference reward led to better overall performance than the standard RLHF objective (Eq. 14), which degraded task completion despite higher preference scores.
• Adaptive scaling of the human preference reward with coefficient $ c $ and proper choice of $ \alpha $ significantly improved task success rates, showing sensitivity to hyperparameter tuning.
• In some hard tasks like Kettle and DoorCloseOutward, the RM could not compensate for poor exploration, and in SwingCup, the lack of human-like behaviors in initial diverse policies made learning difficult, highlighting task-specific limitations.