[Paper Review] NebulOS: A Big Data Framework for Astrophysics

Nathaniel R. Stickley, Miguel A. Aragón-Calvo|arXiv (Cornell University)|Mar 8, 2015

Gamma-ray bursts and supernovae12 references3 citations

TL;DR

NebulOS is a Big Data framework that enables astronomers to run massively parallel data analysis pipelines on datacenter clusters using familiar Python scripts, without modifying existing software. By leveraging Hadoop's HDFS via FUSE-DFS and adding a user-friendly interface and task monitoring, NebulOS reduces development time and enables efficient, fault-tolerant data processing on petascale astrophysical datasets with minimal engineering overhead.

ABSTRACT

We introduce NebulOS, a Big Data platform that allows a cluster of Linux machines to be treated as a single computer. With NebulOS, the process of writing a massively parallel program for a datacenter is no more complicated than writing a Python script for a desktop computer. The platform enables most pre-existing data analysis software to be used, as scale, in a datacenter without modification. The shallow learning curve and compatibility with existing software greatly reduces the time required to develop distributed data analysis pipelines. The platform is built upon industry-standard, open-source Big Data technologies, from which it inherits several fault tolerance features. NebulOS enhances these technologies by adding an intuitive user interface, automated task monitoring, and other usability features. We present a summary of the architecture, provide usage examples, and discuss the system's performance scaling.

Motivation & Objective

• To address the growing challenge of analyzing petascale astrophysical datasets that exceed the I/O and processing capabilities of traditional supercomputers.
• To reduce the software engineering barrier for astronomers by enabling existing data analysis tools to run in a distributed environment without modification.
• To provide a fault-tolerant, scalable, and user-friendly Big Data platform tailored for scientific data workloads in astrophysics.
• To improve usability of Big Data frameworks in astronomy by replacing low-level Java-based programming with a high-level Python interface and automated monitoring.

Proposed method

• NebulOS integrates Hadoop Distributed File System (HDFS) with FUSE (Filesystem in Userspace) to expose HDFS as a local file system, allowing transparent access to distributed data.
• It uses a Python-based application framework that abstracts cluster management, enabling users to write standard Python scripts as if running on a single machine.
• The system includes automated task monitoring and a web-based user interface to simplify cluster operation and debugging.
• It inherits Hadoop’s fault tolerance through data replication and automatic task rescheduling upon node failure.
• The framework supports existing scientific software by allowing direct execution on the cluster without code changes, leveraging HDFS for data locality.
• It enables deployment on cloud platforms like Amazon EC2 via prebuilt system images, allowing rapid cluster provisioning in under 15 minutes.

Experimental results

Research questions

• RQ1 Can a Big Data framework be designed to allow astronomers to run existing analysis software on datacenter clusters without code modification?
• RQ2 How can the usability of Big Data frameworks be improved for non-expert scientific users in astrophysics?
• RQ3 To what extent can data locality and fault tolerance be leveraged to improve performance on petascale astrophysical datasets?
• RQ4 Can a Hadoop-based system be effectively abstracted through FUSE-DFS to provide a seamless local-like experience for scientific workloads?
• RQ5 What are the performance and scalability characteristics of a Python-centric Big Data framework for astronomy data pipelines?

Key findings

• NebulOS enables the execution of existing astrophysical data analysis software on a cluster with no code changes, significantly reducing development time.
• The framework reduces the learning curve by allowing users to write Python scripts as if on a desktop, while transparently scaling to thousands of nodes.
• By using FUSE-DFS to expose HDFS as a local filesystem, NebulOS provides transparent data access and supports data locality, improving I/O performance.
• The system achieves fault tolerance through Hadoop’s native replication and automatic task rescheduling, ensuring reliability in large-scale deployments.
• Deployment on cloud services like Amazon EC2 is streamlined, with a prebuilt system image enabling cluster setup in under 15 minutes.
• Despite limitations with small or large files and lack of native inter-task communication, NebulOS provides a practical, low-overhead solution for data-intensive astrophysical workflows.