[Paper Review] Challenges for LSST scale data sets

M. J. Way|arXiv (Cornell University)|Aug 25, 2011

Astronomy and Astrophysical Research5 references3 citations

TL;DR

This paper examines computational challenges posed by the Large Synoptic Survey Telescope (LSST) data deluge, comparing data access models from 2MASS to SDSS and projecting future workflows. It argues that while local storage of LSST data may be feasible due to declining storage costs, I/O bottlenecks and the O(N²) complexity of advanced analyses will necessitate a shift toward database-local computation, urging the development of robust, high-I/O computational centers for scientific exploitation of LSST-scale datasets.

ABSTRACT

The Large Synoptic Survey Telescope (LSST) simulator being built by Andy Connolly and collaborators is an impressive undertaking and should make working with LSST in the beginning stages far more easy than it was initially with the Sloan Digital Sky Survey. However, I would like to focus on an equally important problem that has not yet been discussed here, but in the coming years the community will need to address -- can we deal with the flood of data from LSST and will we need to rethink the way we work?

Motivation & Objective

• To assess whether traditional data analysis workflows used in 2MASS and SDSS can scale to LSST's petabyte-scale datasets.
• To evaluate the feasibility of local data storage and legacy tool usage for LSST data.
• To investigate whether new computational paradigms—particularly data-local computation—are necessary for handling LSST-scale data.
• To identify the role of high-performance computing centers in enabling scientific discovery from LSST data.

Proposed method

• Compares historical data access models from 2MASS (disk-based, ASCII catalogs) to SDSS (SQL-based, web-query interfaces) to establish a baseline for data interaction.
• Analyzes LSST data scale, estimating a 10 PB query database and 60 PB image data over 10 years, and projects computational demands.
• Applies Amdahl’s rules of thumb to assess system balance, focusing on I/O limitations relative to CPU and memory growth.
• Evaluates the feasibility of running O(N²) algorithms (e.g., clustering, classification) on LSST-scale data, noting that even O(N) algorithms may be impractical without high I/O bandwidth.
• Proposes that only high-I/O computational centers—such as national supercomputing centers—can support the most data-intensive scientific workflows.
• Advocates for the continued use of SQL-based web interfaces (like CASJOBS) but cautions that local data use may be limited by I/O performance.

Experimental results

Research questions

• RQ1 Can researchers continue using traditional 2MASS-style local data analysis tools for LSST-scale datasets despite I/O limitations?
• RQ2 To what extent can SDSS-style web-based SQL querying remain viable for LSST data, given the scale and I/O demands?
• RQ3 Is a fundamental shift to data-local computation necessary for many LSST scientific goals, particularly those involving O(N²) algorithms?
• RQ4 What role do high-I/O computational centers play in enabling scalable analysis of LSST data?
• RQ5 Can historical trends in storage density and I/O performance support local storage and analysis of LSST data?

Key findings

• Local storage of the entire LSST dataset on a desktop system is technically feasible due to declining storage costs and increasing disk densities.
• Despite feasible local storage, sequential I/O bottlenecks—especially at 100MB/s per disk—will severely limit performance when using traditional tools on LSST-scale data.
• A balanced system capable of handling petascale I/O would require approximately 1 million disks each delivering 100MB/s, highlighting a major infrastructure challenge.
• The I/O bottleneck, as per Amdahl’s rules, makes CPU and memory gains insufficient without proportional I/O improvements, rendering many legacy tools impractical for full LSST data analysis.
• Scientific workflows involving O(N²) operations (e.g., clustering, density estimation) will be infeasible on local systems without high-throughput I/O, necessitating computation at centralized data centers.
• National supercomputing centers such as PSC, NCSA, and NAS are best positioned to host LSST data and support expert users requiring data-local computation.