[Paper Review] Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulation

Zhehua Zhou, Jiayang Song+6 more|arXiv (Cornell University)|Jul 31, 2023

Adversarial Robustness in Machine LearningComputer Science61 references3 citations

TL;DR

This paper introduces a public industrial benchmark for AI-enabled robotics manipulation using NVIDIA Isaac Sim, evaluating eight tasks with multiple deep reinforcement learning controllers. It proposes a Python-based falsification framework compatible with physics simulators and OpenAI Gym, demonstrating that state-of-the-art optimization methods effectively expose defects in AI controllers, enhancing reliability assessment for AI-CPS development.

ABSTRACT

As a representative cyber-physical system (CPS), robotic manipulator has been widely adopted in various academic research and industrial processes, indicating its potential to act as a universal interface between the cyber and the physical worlds. Recent studies in robotics manipulation have started employing artificial intelligence (AI) approaches as controllers to achieve better adaptability and performance. However, the inherent challenge of explaining AI components introduces uncertainty and unreliability to these AI-enabled robotics systems, necessitating a reliable development platform for system design and performance assessment. As a foundational step towards building reliable AI-enabled robotics systems, we propose a public industrial benchmark for robotics manipulation in this paper. It leverages NVIDIA Omniverse Isaac Sim as the simulation platform, encompassing eight representative manipulation tasks and multiple AI software controllers. An extensive evaluation is conducted to analyze the performance of AI controllers in solving robotics manipulation tasks, enabling a thorough understanding of their effectiveness. To further demonstrate the applicability of our benchmark, we develop a falsification framework that is compatible with physical simulators and OpenAI Gym environments. This framework bridges the gap between traditional testing methods and modern physics engine-based simulations. The effectiveness of different optimization methods in falsifying AI-enabled robotics manipulation with physical simulators is examined via a falsification test. Our work not only establishes a foundation for the design and development of AI-enabled robotics systems but also provides practical experience and guidance to practitioners in this field, promoting further research in this critical academic and industrial domain.

Motivation & Objective

• To address the lack of systematic evaluation platforms for AI-enabled robotics manipulation in high-fidelity physical simulators.
• To identify industrial and academic needs for AI-CPS development through a global survey of robotics and AI practitioners.
• To establish a public, industrial-grade benchmark in Isaac Sim covering diverse manipulation tasks and AI controllers.
• To develop a flexible, Python-based falsification framework compatible with physics engines and OpenAI Gym environments.
• To evaluate the effectiveness of optimization-based falsification in uncovering vulnerabilities in AI controllers for robotics.

Proposed method

• Conducted a global survey with robotics and AI practitioners to identify key requirements and limitations of Isaac Sim compared to other simulators.
• Designed a benchmark with eight representative robotics manipulation tasks in NVIDIA Omniverse Isaac Sim, including pick-and-place, insertion, and stacking.
• Trained multiple AI controllers using deep reinforcement learning algorithms and evaluated their performance across tasks and under action noise.
• Developed a falsification framework in Python that integrates with physics-based simulators and OpenAI Gym environments for automated defect discovery.
• Employed three optimization methods—CMA-ES, SLSQP, and Bayesian optimization—for falsification to test controller robustness.
• Executed falsification tests to assess the ability of these methods to generate adversarial scenarios that cause controller failure.

Experimental results

Research questions

• RQ1 How do different AI controllers perform across a diverse set of industrial robotics manipulation tasks in a high-fidelity simulator?
• RQ2 To what extent can physics engine-based simulators like Isaac Sim accurately reflect real-world system behavior for AI-CPS development?
• RQ3 How effective are state-of-the-art optimization-based falsification techniques in identifying vulnerabilities in AI controllers for robotics?
• RQ4 What are the key requirements and limitations of current simulators from the perspective of robotics and AI practitioners?
• RQ5 Can a unified framework bridge traditional testing methods with modern physics-based simulation for AI-CPS validation?

Key findings

• AI controllers trained with deep reinforcement learning successfully completed all eight benchmark tasks with high success rates and demonstrated robustness under action noise.
• The falsification framework successfully identified failure scenarios for AI controllers using optimization-based search, confirming its utility in reliability testing.
• CMA-ES, SLSQP, and Bayesian optimization all proved effective in generating adversarial inputs that triggered controller failures, with varying efficiency across tasks.
• The survey revealed that practitioners value high-fidelity physics simulation, GPU acceleration, and seamless integration with AI training pipelines—key strengths of Isaac Sim.
• The benchmark and falsification framework are publicly available, enabling reproducibility and future research in AI-CPS validation.
• The study establishes a systematic pipeline for evaluating and testing AI-enabled robotics systems in industrial-grade simulation environments.