[论文解读] A Study on the Challenges of Using Robotics Simulators for Testing

Afsoon Afzal, Deborah Katz+2 more|arXiv (Cornell University)|Apr 15, 2020

Software Testing and Debugging Techniques参考文献 24被引用 19

一句话总结

本研究基于对82名机器人开发者的调查，探讨了机器人开发者在使用仿真进行测试和测试自动化时面临的挑战。研究识别出10项关键挑战——从真实感差距和可复现性问题，到高资源成本和自动化支持不足——并提出下一代仿真器必须提升易用性、可扩展性，并与CI/CD流水线更好地集成，以充分发挥仿真在机器人验证与确认中的潜力。

ABSTRACT

Robotics simulation plays an important role in the design, development, and verification and validation of robotic systems. Recent studies have shown that simulation may be used as a cheaper, safer, and more reliable alternative to manual, and widely used, process of field testing. This is particularly important in the context of continuous integration pipelines, where integrated automated testing is key to reducing costs while maintaining system safety. However, simulation and automated testing are not seeing the degree of widespread adoption in practice that their potential would motivate. Our goal in this paper is to develop a principled understanding of the ways developers use simulation in their process, and the challenges they face in doing so. This type of understanding can guide the development of more effective simulators and testing techniques for modern robotics development. To that end, we conduct a survey of 82 robotics developers from a diversity of backgrounds that addresses the current capabilities and limits of simulation technology in practice. We find that simulation is used by 85% of our participants for testing, and that many participants desire to use simulation as part of their test automation. We identify 10 high-level challenges that impede developers from using simulation for manual and automated testing, and general purposes. These challenges include the gap between simulation and reality, a lack of reproducibility, and considerable resource costs associated with using simulators. Finally, we outline avenues for improvement in the development of new simulators that can help simulation reach its potential as a means of verification and validation.

研究动机与目标

• 了解机器人开发者当前在开发工作流中如何使用仿真。
• 识别阻碍仿真在测试和测试自动化中广泛采用的主要挑战。
• 基于真实开发者需求和痛点，为下一代仿真器的设计提供依据。
• 指导开发更易用、更可靠、更可扩展的仿真工具，以支持机器人验证与确认。

提出的方法

• 对来自不同组织和经验水平的82名机器人开发者开展了在线开放式调查。
• 收集了在通用使用、测试和测试自动化场景下，仿真使用模式、感知挑战及期望改进的信息。
• 通过主题分析识别并分类影响仿真采用的10项高层次挑战。
• 评估了现有仿真器（如CARLA、AirSim、Ignition Gazebo、AWS RoboMaker），以基准对比当前能力与识别出的挑战。
• 基于开发者反馈和对现有工具的分析，提出了未来仿真器的设计原则。

实验结果

研究问题

• RQ1 开发者在通用场景下使用仿真时面临哪些挑战？
• RQ2 开发者在使用仿真进行测试时面临哪些挑战？
• RQ3 开发者在使用仿真进行测试自动化时面临哪些挑战？
• RQ4 当前仿真器在真实感、易用性和自动化支持方面与开发者需求相比如何？

主要发现

• 85%的受访开发者使用仿真进行测试，60%将其作为测试自动化的一部分，表明已有较强的采用基础。
• 主要挑战包括仿真与真实世界行为之间的显著差距、缺乏可复现性，以及运行仿真所需的高资源成本。
• 开发者反映仿真器易用性存在问题，包括陡峭的学习曲线、文档质量差，以及缺乏用户友好的功能（如Web界面）。
• 自动化受到不稳定API、缺乏无头执行支持，以及不可靠或非确定性仿真结果的阻碍。
• 许多开发者希望仿真器能在不带来过高计算成本的前提下，支持复杂且大规模的环境，并具备高物理和视觉保真度。
• 本研究提出三大设计核心原则：提升易用性，增强真实感与可扩展性，并为CI/CD流水线提供强大且低成本的自动化支持。