[Paper Review] Evaluating Real-World Robot Manipulation Policies in Simulation

Xuanlin Li, Kyle Hsu+14 more|arXiv (Cornell University)|May 9, 2024

Robot Manipulation and Learning4 citations

TL;DR

This paper proposes SIMPLER, a suite of scalable, reproducible, and reliable simulated environments for evaluating real-world robot manipulation policies without requiring full-fidelity digital twins. By addressing control and visual disparities through system identification, green-screening, and texture baking, the authors demonstrate strong correlation between simulated and real-world performance across diverse policies and tasks, enabling effective sim-to-real evaluation for generalist manipulation policies.

ABSTRACT

The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks.

Motivation & Objective

• Address the scalability and reproducibility challenges in real-world evaluation of generalist robot manipulation policies.
• Overcome the limitations of full-fidelity digital twins, which are time-consuming and resource-intensive to build.
• Develop a simulation-based evaluation framework that maintains strong correlation with real-world performance despite control and visual discrepancies.
• Enable accurate reflection of policy behavior modes, such as sensitivity to distribution shifts, in simulation.
• Facilitate broader research in generalist manipulation policies by open-sourcing SIMPLER environments and evaluation workflows.

Proposed method

• Use offline system identification to calibrate control dynamics between real and simulated environments, reducing control disparities.
• Apply 'green-screening' to simulate observations using real-world backgrounds, improving visual realism and alignment.
• Bake object textures from real-world images to enhance visual fidelity in simulation without requiring full 3D reconstruction.
• Design a modular pipeline for creating new SIMPLER environments using standard Gym interfaces, enabling easy integration and reuse.
• Evaluate policies in SIMPLER environments and compare results directly with real-world performance using metrics like Pearson correlation and MMRV.
• Conduct Kruskal-Wallis tests to assess distributional shifts in absolute success rates between real and simulated evaluations.

Figure 0 : We introduce SIMPLER, a suite of open-source simulated evaluation environments for common real robot manipulation setups, namely the Google Robot evaluations from the RT-series of works [ 6 , 5 , 11 ] , and environments from the BridgeData V2 dataset [ 66 ] . All environments can be impor

Experimental results

Research questions

• RQ1 Can simulation-based evaluation serve as a scalable, reproducible, and reliable proxy for real-world evaluation of generalist manipulation policies?
• RQ2 To what extent do control and visual disparities between real and simulated environments degrade the fidelity of simulated policy evaluation?
• RQ3 Can effective simulated evaluation be achieved without constructing full-fidelity digital twins of real environments?
• RQ4 Does SIMPLER accurately reflect real-world policy behavior modes, such as robustness to visual distribution shifts?
• RQ5 How well does SIMPLER generalize across policies trained on diverse data scales, including single-task policies?

Key findings

• SIMPLER evaluation shows a substantially higher Pearson correlation coefficient and significantly lower MMRV (Mean Median Relative Variance) between real and simulated performance compared to baseline simulation methods.
• The framework accurately reflects policy behavior modes, such as sensitivity to visual distribution shifts, demonstrating fidelity beyond simple performance ranking.
• Even for single-task policies like 'pick coke can', SIMPLER maintains low MMRV and high Pearson correlation, confirming its effectiveness across data scales.
• Kruskal-Wallis tests show that simulation success distributions are not significantly different from real-world distributions in most tasks, indicating minimal absolute performance shift.
• The Visual Matching evaluation approach results in no significant distribution shifts (p ≥ 0.05) for most tasks, validating the robustness of the simulation pipeline.
• The combination of MMRV and Kruskal-Wallis metrics provides complementary insights: MMRV measures relative performance alignment, while Kruskal-Wallis assesses absolute distributional consistency.

Figure 1 : Illustration of Mean Maximum Rank Violation (MMRV, range $[0,1]$ , lower is better) and Pearson correlation coefficient (Pearson $r$ , range $[-1,1]$ , higher is better) for assessing the correlation between policy performances in real-world and simulation, as well as the overall quality