[Paper Review] Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources

Logan Ward, J. Gregory Pauloski+8 more|arXiv (Cornell University)|Mar 15, 2023

Scientific Computing and Data ManagementDecision Sciences45 references3 citations

TL;DR

This paper presents a cloud-native, FaaS-based workflow framework using FuncX and Globus to enable efficient, secure, and portable AI-guided simulation workflows across heterogeneous HPC and GPU resources. By leveraging pass-by-reference data transfer and user-configurable steering, the approach achieves scientific performance parity with traditional direct-connection workflows while simplifying deployment and reducing operational complexity.

ABSTRACT

Applications that fuse machine learning and simulation can benefit from the use of multiple computing resources, with, for example, simulation codes running on highly parallel supercomputers and AI training and inference tasks on specialized accelerators. Here, we present our experiences deploying two AI-guided simulation workflows across such heterogeneous systems. A unique aspect of our approach is our use of cloud-hosted management services to manage challenging aspects of cross-resource authentication and authorization, function-as-a-service (FaaS) function invocation, and data transfer. We show that these methods can achieve performance parity with systems that rely on direct connection between resources. We achieve parity by integrating the FaaS system and data transfer capabilities with a system that passes data by reference among managers and workers, and a user-configurable steering algorithm to hide data transfer latencies. We anticipate that this ease of use can enable routine use of heterogeneous resources in computational science.

Motivation & Objective

• To address the growing need for efficient, secure, and portable execution of AI-guided simulation workflows across heterogeneous computing resources.
• To reduce deployment complexity and eliminate single points of failure in multi-resource scientific workflows.
• To minimize data transfer latency and overhead in AI and simulation workloads through intelligent steering and efficient data movement.
• To demonstrate that cloud-managed services can achieve performance parity with conventional, direct-connection workflow systems.
• To enable broader adoption of heterogeneous computing in computational science by simplifying secure, scalable, and reliable cross-resource orchestration.

Proposed method

• Employing a federated Function-as-a-Service (FaaS) platform (FuncX) to manage remote task execution across HPC and GPU resources without direct resource connections.
• Using ProxyStore to enable peer-to-peer, Globus-based data transfer via reference, minimizing data movement through the central controller.
• Integrating Colmena to express user-configurable steering policies that hide data transfer latencies by overlapping computation and I/O.
• Passing task instructions and data by reference to reduce controller load and avoid bottlenecks in large-scale, data-intensive workflows.
• Leveraging cloud-hosted coordination services for authentication, authorization, and error handling to improve reliability and reduce deployment complexity.
• Benchmarking performance using synthetic workloads and two real-world applications: molecular design and surrogate model training.

Experimental results

Research questions

• RQ1 Can cloud-hosted FaaS and data transfer services achieve performance parity with traditional, direct-connection workflow systems in AI-guided simulation workloads?
• RQ2 How effective are pass-by-reference data transfer and intelligent steering in reducing end-to-end latency in multi-resource scientific workflows?
• RQ3 What is the impact of data size and transfer mechanism (e.g., Redis vs. Globus) on task overhead and workflow performance?
• RQ4 To what extent can cloud-managed services simplify deployment and improve reliability in heterogeneous, multi-site scientific computing environments?
• RQ5 Can the integration of FaaS, data reference passing, and dynamic steering enable scalable, secure, and portable AI-simulation workflows without sacrificing scientific accuracy?

Key findings

• The surrogate models generated using the cloud-managed workflow (FuncX + Globus) had an RMSD of 1.30 ± 0.08 eV/Å, comparable to the conventional Parsl-based approach (1.47 ± 0.09 eV/Å), demonstrating scientific performance parity.
• Task overhead for GPU tasks was dominated by Globus data transfer times, averaging approximately 2 seconds per direction, consistent with synthetic benchmark results.
• For CPU tasks, FuncX overhead was primarily due to notification latency to the controller, while Parsl’s overhead scaled with data size—820 ms for 3 MB messages and 20 ms for 20 kB messages.
• Pass-by-reference reduced data transfer overhead significantly for larger messages, with consistent transfer times of 200 ms (sampling) and 170 ms (simulation) in Parsl with reference passing.
• The use of pass-by-reference and intelligent steering enabled up to 10x reductions in application latency compared to naive data transfer approaches.
• For data larger than 10 kB, direct transfer via Globus or Redis (if feasible) was more efficient than proxying through the controller, with Redis being faster when direct tunnels were available.