[Paper Review] The LAMOST Data Archive and Data Release

Boliang He, Dongwei Fan+4 more|arXiv (Cornell University)|Jan 11, 2016

Advanced Data Storage Technologies3 citations

TL;DR

This paper presents the LAMOST Data Archive and Data Release system, detailing a robust, four-version-evolved data cycle management framework that handles raw data transfer, pipeline processing, archiving, backup, and public release. The system successfully delivered four data releases (PDR, DR1–DR3) over four years, managing 18TB of raw data and 4 million spectra, enabling large-scale stellar spectroscopy research through a scalable, production-ready infrastructure built on open-source tools and China-VO integration.

ABSTRACT

The Large sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) is the largest optical telescope in China. In last four years, the LAMOST telescope has published four editions data (pilot data release, data release 1, data release 2 and data release 3). To archive and release these data (raw data, catalog, spectrum etc), we have set up a data cycle management system, including the transfer of data, archiving, backup. And through the evolution of four software versions, mature established data release system.

Motivation & Objective

• To establish a reliable, scalable data cycle management system for the LAMOST telescope’s massive spectroscopic data output.
• To ensure long-term preservation and public accessibility of raw data, catalogues, and FITS spectra through automated archiving and backup.
• To support the scientific community by releasing multiple editions of processed data (PDR, DR1, DR2, DR3) with consistent quality and metadata.
• To integrate with China-VO and leverage open-source software stacks for sustainable, extensible data infrastructure.

Proposed method

• Implement a multi-stage data flow: raw data transferred from LAMOST site to China-VO Data Center, then pushed to a Pipeline Server for product generation.
• Use a Spring Framework-based web application to manage data release workflows and user access via CSTNET Passport.
• Store processed data (catalogs and spectra) in PostgreSQL with pgSphere extensions, hosted on a dedicated Data Release Server.
• Back up both raw and product data on a third-party storage system to ensure data durability and disaster recovery.
• Deploy Nginx as the web server to serve data release content and manage traffic efficiently.
• Manage software development and versioning through a China-VO-hosted GitLab-based code repository system.

Experimental results

Research questions

• RQ1 How can a large-scale, high-throughput data pipeline be designed for a multi-object fiber spectroscopic survey like LAMOST?
• RQ2 What system architecture ensures reliable data transfer, processing, archiving, and public release of petabyte-scale astronomical data?
• RQ3 How can open-source software and national data infrastructure (China-VO) be integrated to support sustainable data management for a national scientific facility?
• RQ4 What are the key performance and scalability metrics of a data release system handling 18TB of raw data and 4 million spectra over four years?

Key findings

• The LAMOST Data Archive and Data Release system successfully managed 900 observation nights, producing approximately 18TB of raw data and 5TB of product data (catalogs and FITS spectra).
• Four major data releases—Pilot Data Release (PDR), DR1, DR2, and DR3—were issued between 2012 and 2015, with DR1 and DR2 publicly released in 2013 and 2014, respectively.
• The average raw data size per night was approximately 20GB, consistent across the 900 observation nights.
• The system achieved stable, automated data processing and release using a modular, four-version-evolved software stack based on open-source technologies including Spring, PostgreSQL, Nginx, and GitLab.
• The integration with China-VO enabled secure, scalable, and publicly accessible data dissemination through dedicated web portals (e.g., http://dr3.lamost.org).
• The data release pipeline reliably produced and delivered 4 million spectra and associated catalogues, supporting large-scale stellar population studies.