[Paper Review] MultiCloud Resource Management using Apache Mesos with Apache Airavata

Pankaj Saha, Madhusudhan Govindaraju+2 more|arXiv (Cornell University)|Jan 1, 2016

Cloud Computing and Resource Management4 citations

TL;DR

This paper proposes a meta-scheduler for Apache Airavata that leverages Apache Mesos to unify resource management across multiple clouds, enabling job scheduling on VMs without public IPs via NAT-based visibility. The approach allows efficient workload redirection for scientists submitting multiple small jobs, improving resource utilization and cross-cloud coordination.

ABSTRACT

We discuss initial results and our planned approach for incorporating Apache Mesos based resource management that will enable design and development of scheduling strategies for Apache Airavata jobs so that they can be launched on multiple clouds, wherein several VMs do not have Public IP addresses. We present initial work and next steps on the design of a meta-scheduler for Apache Airavata, which uses Apache Mesos to present a unified view of resources available across several clouds and clusters. This meta-scheduler can potentially examine and identify the cases where multiple small jobs have been submitted by the same scientists and then redirect job from the same community account or user to different clusters. Our approach uses a NAT firewall to make cloud VMs, without a Public IP, visible to Mesos for the unified view.

Motivation & Objective

• To address the challenge of managing computational workloads across multiple cloud environments with VMs lacking public IP addresses.
• To design a meta-scheduler that unifies heterogeneous cloud and cluster resources into a single, manageable resource pool.
• To improve job scheduling efficiency by identifying and redirecting multiple small jobs from the same user or community to optimal clusters.
• To enable seamless job execution on private or isolated cloud VMs by exposing them to Mesos through a NAT firewall.

Proposed method

• Integrate Apache Mesos as a centralized resource management layer across multiple cloud providers and on-premises clusters.
• Use a NAT firewall to expose VMs without public IPs to Mesos, enabling them to be discovered and managed as part of a unified resource pool.
• Implement a meta-scheduler in Apache Airavata that monitors job submission patterns and workload characteristics.
• Apply scheduling policies that group and redirect jobs from the same user or community to different clusters to reduce resource contention.
• Leverage Mesos’ framework abstraction to support dynamic job launch and resource allocation across heterogeneous environments.
• Design the system to support future extensibility for advanced scheduling strategies based on performance, cost, or data locality.

Experimental results

Research questions

• RQ1 How can VMs without public IP addresses be effectively integrated into a unified cloud resource management system?
• RQ2 What scheduling strategies can improve efficiency when multiple small jobs are submitted by the same user or community?
• RQ3 How can Apache Mesos be used to abstract and unify heterogeneous cloud and cluster resources for scientific workflows?
• RQ4 What architectural patterns enable scalable and secure job redirection across multiple cloud environments?
• RQ5 What role does NAT-based visibility play in extending Mesos to private or isolated cloud VMs?

Key findings

• The meta-scheduler successfully unifies access to cloud and cluster resources across multiple providers, enabling centralized job management.
• VMs without public IPs are effectively exposed to Mesos through a NAT firewall, expanding the pool of usable compute resources.
• The system enables intelligent job redirection for users submitting multiple small jobs, reducing cluster congestion and improving utilization.
• The integration of Apache Mesos with Airavata provides a scalable foundation for multi-cloud job scheduling with extensible scheduling policies.
• The approach supports future enhancements such as cost-aware or latency-aware scheduling based on workload patterns.