[Paper Review] The Data Lab: A Science Platform for the analysis of ground-based astronomical survey data

Knut Olsen, A. Bolton+7 more|arXiv (Cornell University)|Aug 1, 2019

Astronomy and Astrophysical Research4 citations

TL;DR

The NOAO Data Lab is a science platform designed to enable efficient analysis of petabyte-scale ground-based astronomical survey data, leveraging standardized protocols, open-source tools, and cloud-ready containerization. It supports large-scale catalog and image queries, with 864 registered users and over 6,000 daily automated queries, demonstrating its role in democratizing access to massive survey datasets for discovery-driven science.

ABSTRACT

The next decade will feature a growing number of massive ground-based photometric, spectroscopic, and time-domain surveys, including those produced by DECam, DESI, and LSST. The NOAO Data Lab was launched in 2017 to enable efficient exploration and analysis of large surveys, with particular focus on the petabyte-scale holdings of the NOAO Archive and their associated catalogs. The Data Lab mission and future development align well with two of the NSF's Big Ideas, namely Harnessing Data for 21st Century Science and Engineering and as part of a network to contribute to Windows on the Universe: The Era of Multi-messenger Astrophysics. Along with other Science Platforms, the Data Lab will play a key role in scientific discoveries from surveys in the next decade, and will be crucial to maintaining a level playing field as datasets grow in size and complexity.

Motivation & Objective

• To address the growing challenge of analyzing petabyte-scale astronomical survey data from NOAO telescopes, including DECam, DESI, and future LSST surveys.
• To provide a science platform that supports interactive exploration and automated analysis of catalogs, images, and spectra, ensuring broad community access.
• To maintain a level playing field for researchers by offering scalable infrastructure for data-intensive science as datasets grow in size and complexity.
• To align with NSF Big Ideas in data science and multi-messenger astrophysics by integrating with national science platforms and data ecosystems.
• To ensure long-term sustainability and interoperability through adoption of IVOA standards, open-source software, and containerized deployment.

Proposed method

• The platform uses IVOA-compliant protocols such as Table Access Protocol (TAP), Simple Image Access (SIA), VOSpace, and Universal Worker Service (UWS) for data discovery and access.
• It integrates open-source software stacks including PostgreSQL for catalog storage, Q3C for spatial indexing and cone searches, and Jupyter for interactive analysis.
• The system leverages AstroPy and affiliated packages for scientific computing, and PyVO for VO-compliant data access and interoperability.
• Data Lab employs Docker and Kubernetes for containerized deployment and service orchestration, enabling scalable and reproducible scientific workflows.
• It supports automated, scripted queries through a high-throughput query pipeline, with 50% of daily queries being image cutouts and 45% targeting DECam and Mosaic catalogs.
• The platform is designed for future compatibility with the LSST Science Platform and is part of a broader Science Platform Network to ensure interoperability across national facilities.

Experimental results

Research questions

• RQ1 How can a science platform efficiently support the analysis of petabyte-scale astronomical survey data from ground-based telescopes?
• RQ2 What infrastructure and software stack are required to enable scalable, interactive, and automated analysis of large catalogs, images, and spectra?
• RQ3 How can standardized, open-protocol services ensure interoperability and long-term sustainability in a multi-platform data ecosystem?
• RQ4 What role do science platforms play in maintaining equitable access to large datasets across the astronomical research community?
• RQ5 How can containerization and cloud-ready deployment strategies improve the maintainability and extensibility of scientific data platforms?

Key findings

• As of the paper’s publication, the Data Lab had 864 registered users, with user numbers growing linearly since its June 2017 public release.
• The platform processed an average of over 6,000 queries per day over an eight-month period, the majority of which were automated and scripted.
• Approximately 50% of daily queries were for image cutouts, while 45% targeted DECam- and Mosaic-based catalogs, indicating strong demand for imaging and catalog data.
• Interactive usage, measured by catalog cross-matches and Jupyter notebook access, accounted for only about 5% of queries, confirming that most user engagement is automated.
• The platform is on a continuous incremental release cycle, with version 2.17.1 in active use, reflecting a shift from monolithic releases to agile, frequent updates.
• The annual operational cost is approximately $1 million, with 7.5 FTEs dedicated to development and operations, and $115,000 allocated to equipment and travel.