[Paper Review] Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation

Niklas Funk, Charles Schaff+11 more|arXiv (Cornell University)|May 5, 2021

Robot Manipulation and Learning53 references23 citations

TL;DR

This paper benchmarks structured policies and optimization techniques for real-world dexterous manipulation using the TriFinger robotic platform. It proposes motion planning, compliant impedance control, and a residual policy learning approach, demonstrating that Bayesian optimization significantly improves hyperparameter tuning and that residual policies enhance robustness—especially for motion planning—while enabling successful sim-to-real transfer without domain randomization.

ABSTRACT

Dexterous manipulation is a challenging and important problem in robotics. While data-driven methods are a promising approach, current benchmarks require simulation or extensive engineering support due to the sample inefficiency of popular methods. We present benchmarks for the TriFinger system, an open-source robotic platform for dexterous manipulation and the focus of the 2020 Real Robot Challenge. The benchmarked methods, which were successful in the challenge, can be generally described as structured policies, as they combine elements of classical robotics and modern policy optimization. This inclusion of inductive biases facilitates sample efficiency, interpretability, reliability and high performance. The key aspects of this benchmarking is validation of the baselines across both simulation and the real system, thorough ablation study over the core features of each solution, and a retrospective analysis of the challenge as a manipulation benchmark. The code and demo videos for this work can be found on our website (https://sites.google.com/view/benchmark-rrc).

Motivation & Objective

• To establish a reproducible benchmark for real-world dexterous manipulation using the open-source TriFinger platform.
• To evaluate structured policies that combine classical robotics with modern reinforcement learning for improved sample efficiency and reliability.
• To investigate the effectiveness of hyperparameter optimization via Bayesian optimization and residual policy learning in real-world settings.
• To analyze the sim-to-real transfer performance of learned policies and identify key components enabling robust real-world deployment.
• To provide a retrospective analysis of the 2020 Real Robot Challenge to inform future benchmarking and competition design.

Proposed method

• Developed three structured control policies: motion planning (MP), compliant impedance control (CIC), and a task-specific grasp policy (CPC) using torque commands.
• Applied Bayesian optimization (BO) to tune hyperparameters of each controller, using Gaussian process models to guide search over the objective function.
• Integrated residual policy learning (RPL) to refine base controllers by learning corrective actions via reinforcement learning.
• Conducted extensive ablation studies in simulation and on real TriFinger robots, evaluating performance across multiple task variants and goal distributions.
• Used a standardized evaluation protocol with metrics including sparse dense reward, final position error, and drop rate across 20 rollouts per condition.
• Evaluated transfer performance from simulation to real hardware without domain randomization or fine-tuning, assessing robustness and generalization.

Experimental results

Research questions

• RQ1 How do structured policies combining classical control and learned components compare in accuracy, reliability, and sample efficiency on real-world dexterous manipulation tasks?
• RQ2 To what extent can Bayesian optimization improve controller performance by tuning hyperparameters across different policy structures?
• RQ3 Does residual policy learning enhance the robustness of base controllers in real-world settings, and does it enable successful sim-to-real transfer without domain randomization?
• RQ4 How do performance characteristics vary between simulation and real-world deployment, and what factors contribute to the sim-to-real gap?
• RQ5 What insights from the 2020 Real Robot Challenge can inform the design of future benchmarks and competition protocols for dexterous manipulation?

Key findings

• Bayesian optimization significantly improved performance across all controllers on the L3 task, increasing mean sparse dense reward and reducing position errors in both simulation and real-world evaluation.
• For the MP-PG controller, residual policy learning reduced the drop rate from 40.0% to 1.74% on the real system and improved grasp stability, preventing catastrophic failures.
• The CPC-TG controller achieved the highest reward in simulation (−70.0 ± 9.42) and low drop rate (0.25 ± 0.15), but the MP-PG controller won the competition due to greater reliability in real-world execution.
• Surprisingly, residual policies transferred directly from simulation to real hardware without fine-tuning or domain randomization, indicating strong generalization capability.
• The CIC-CG controller showed improved performance with BO, achieving a mean reward of −570 ± 350.6 and a drop rate of 3.52 ± 3.91 in simulation, with minimal degradation in real-world performance.
• Even without manual hyperparameter priors, BO optimized parameters that matched or exceeded manually tuned performance, demonstrating its effectiveness in real-world control optimization.