[Paper Review] Batch-oriented software appliances

Riccardo Murri, Sergio Maffioletti|arXiv (Cornell University)|Mar 7, 2012

Cloud Computing and Resource Management5 references3 citations

TL;DR

AppPot is a batch-oriented software appliance system that enables users to create and deploy customized Linux virtual appliances as regular batch jobs on grid or cluster infrastructures without requiring root access or special virtualization support. By leveraging User-Mode Linux (UML), AppPot allows users to package applications with their full runtime environment, enabling portable, reproducible, and self-contained execution across heterogeneous computing environments including private clouds, public clouds, and traditional batch systems.

ABSTRACT

This paper presents AppPot, a system for creating Linux software appliances. AppPot can be run as a regular batch or grid job and executed in user space, and requires no special virtualization support in the infrastructure. The main design goal of AppPot is to bring the benefits of a virtualization-based IaaS cloud to existing batch-oriented computing infrastructures. In particular, AppPot addresses the application deployment and configuration on large heterogeneous computing infrastructures: users are enabled to prepare their own customized virtual appliance for providing a safe execution environment for their applications. These appliances can then be executed on virtually any computing infrastructure being in a private or public cloud as well as any batch-controlled computing clusters the user may have access to. We give an overview of AppPot and its features, the technology that makes it possible, and report on experiences running it in production use within the Swiss National Grid infrastructure SMSCG.

Motivation & Objective

• Address the challenge of deploying complex, custom scientific applications in large-scale, heterogeneous computing infrastructures where centralized software management is impractical.
• Overcome the limitations of traditional batch systems, which often restrict software installation and require administrative intervention for application setup.
• Enable users to create and run their own software appliances that encapsulate complete runtime environments, ensuring reproducibility and isolation.
• Facilitate seamless transition between development (on local machines) and execution (in batch or cloud environments) using identical virtual appliances.
• Reduce administrative overhead by allowing users to self-deploy applications without requiring system administrator involvement or privileged access.

Proposed method

• Use POSIX shell scripts to modify the boot sequence of a Linux system image, enabling it to be launched as a batch job via the kernel command line.
• Integrate with User-Mode Linux (UML), a lightweight virtualization technology that runs Linux kernels as user-space processes, enabling virtualization without hardware support or root privileges.
• Provide a command-line utility, `apppot-start`, to invoke an AppPot image and execute a specified command within the virtualized environment.
• Package applications and their dependencies into a minimal, self-contained Linux image that includes the operating system, libraries, and runtime configuration.
• Leverage UML’s ability to run as a regular process, allowing integration with existing batch schedulers (e.g., SGE, SLURM) and grid middleware.
• Support portability by enabling the same appliance to be used across local clusters, private clouds, and public IaaS environments.

Experimental results

Research questions

• RQ1 Can a lightweight, user-space virtualization approach enable the deployment of complex scientific applications in batch-oriented computing infrastructures without requiring system administrator intervention?
• RQ2 To what extent can User-Mode Linux be used to create portable, self-contained software appliances that function as batch jobs in grid and cluster environments?
• RQ3 How effectively can AppPot reduce the administrative burden of software deployment while maintaining compatibility with existing batch system workflows?
• RQ4 What are the performance and scalability limitations of running UML-based appliances in memory-constrained batch environments?
• RQ5 Can the same software appliance be used consistently across development, testing, and production environments in distributed computing infrastructures?

Key findings

• AppPot successfully enables users to package and deploy complex scientific applications as self-contained software appliances that can be executed as batch jobs without requiring root access or infrastructure-level virtualization support.
• The system has been deployed in production within the Swiss National Grid Infrastructure (SMSCG), demonstrating feasibility and usability in real-world large-scale computing environments.
• AppPot appliances are portable and can be executed identically on local clusters, private clouds, and public IaaS clouds, ensuring reproducibility and consistency across execution environments.
• The use of UML allows AppPot to run in user space, avoiding the need for privileged operations or hypervisor-level virtualization, which simplifies deployment and reduces administrative overhead.
• A known limitation is that UML’s memory accounting mechanism incorrectly estimates shared memory pages, causing batch systems to prematurely kill jobs due to perceived memory overuse, though this can be mitigated by disabling memory enforcement for UML jobs.
• While UML lacks native SMP support, limiting multi-threaded application performance, future extensions could enable MPI-based parallel execution via UDP multicast tunneling, enhancing support for HPC-style workloads.