[Paper Review] CANFAR+Skytree: A Cloud Computing and Data Mining System for Astronomy

Nicholas M. Ball|arXiv (Cornell University)|Dec 14, 2013

Astronomical Observations and Instrumentation3 citations

TL;DR

This paper presents CANFAR+Skytree, the world’s first cloud-based data mining system for astronomy, integrating CANFAR’s scalable cloud infrastructure with Skytree’s high-performance machine learning algorithms. It enables large-scale, linearly scalable data mining on astronomical datasets—demonstrated by computing photometric redshift probability density functions for a 13-billion-object catalog from the CFHTLS survey, achieving full-sophistication analysis on petascale data.

ABSTRACT

At the Canadian Astronomy Data Centre, we have combined our cloud computing system, CANFAR, with the world's most advanced machine learning software, Skytree, to create the world's first cloud computing system for data mining in astronomy. CANFAR provides a generic environment for the storage and processing of large datasets, removing the requirement to set up and maintain a computing system when implementing an extensive undertaking such as a survey pipeline. 500 processor cores and several hundred terabytes of persistent storage are currently available to users. The storage is implemented via the International Virtual Observatory Alliance's VOSpace protocol, and is accessible both interactively, and to all processing jobs. The user interacts with CANFAR by utilizing virtual machines, which appear to them as equivalent to a desktop. Each machine is replicated as desired to perform large-scale parallel processing. Such an arrangement enables the user to immediately install and run the same astronomy code that they already utilize, in the same way as on a desktop. In addition, unlike many cloud systems, batch job scheduling is handled for the user on multiple virtual machines by the Condor job queueing system. Skytree is installed and run just as any other software on the system, and thus acts as a library of command line data mining functions that can be integrated into one's wider analysis. Thus we have created a generic environment for large-scale analysis by data mining, in the same way that CANFAR itself has done for storage and processing. Because Skytree scales to large data in linear runtime, this allows the full sophistication of the huge fields of data mining and machine learning to be applied to the hundreds of millions of objects that make up current large datasets. We demonstrate the utility of the CANFAR+Skytree system by showing science results obtained. [Abridged]

Motivation & Objective

• Address the gap between sophisticated analysis of small datasets and simple analysis of large datasets in astronomy.
• Enable astronomers to perform advanced data mining on massive datasets without requiring expertise in machine learning or infrastructure management.
• Provide a scalable, cloud-based platform that combines CANFAR’s storage and processing with Skytree’s high-performance machine learning algorithms.
• Demonstrate the system’s capability on real astronomical data, particularly for photometric redshift estimation using full probability density functions.
• Create an open, accessible system for the broader astronomical community to perform large-scale data mining on upcoming survey data, such as from the LSST.

Proposed method

• Leverage CANFAR’s cloud infrastructure, providing 500 processor cores and several hundred terabytes of VOSpace-compliant persistent storage accessible as a mounted filesystem.
• Use virtual machines to emulate desktop environments, allowing users to run existing astronomy software unchanged in a scalable, parallelized cloud setting.
• Integrate Skytree as a command-line library of machine learning functions, enabling seamless inclusion in data processing pipelines.
• Utilize the Condor job queueing system to automate batch job scheduling across multiple virtual machines, abstracting low-level parallelization.
• Apply Skytree’s optimized algorithms—such as allkn (k-nearest neighbors) and kde (kernel density estimation)—that scale linearly with dataset size, avoiding quadratic complexity.
• Process large astronomical catalogs by distributing computation across CANFAR’s scalable compute nodes, ensuring high performance and reproducibility.

Experimental results

Research questions

• RQ1 Can a cloud-based system effectively combine scalable storage and high-performance data mining for astronomical datasets?
• RQ2 Can Skytree’s linearly scaling machine learning algorithms be practically deployed in an astronomy cloud environment to analyze petascale data?
• RQ3 Can full probability density functions for photometric redshifts be efficiently computed on large catalogs using this system?
• RQ4 Does the integration of CANFAR and Skytree enable non-specialists to perform advanced data mining on large astronomical datasets?
• RQ5 Can this system handle datasets of the scale expected from next-generation surveys like the LSST?

Key findings

• The CANFAR+Skytree system successfully enabled the computation of photometric redshift probability density functions (PDFs) for 13 billion object instances from the CFHTLS survey, demonstrating full-sophisticated analysis on petascale data.
• Skytree’s allkn algorithm achieved linear runtime scaling with dataset size, confirming its ability to scale to large astronomical datasets without quadratic performance degradation.
• Performance benchmarks showed runtime error bars were smaller than the data points, indicating high reproducibility and stability, especially on high-memory nodes (256GB).
• The system allowed astronomers to run their existing code in a cloud environment indistinguishable from a desktop, with automatic parallelization via Condor.
• The integration of Skytree into CANFAR’s infrastructure enabled non-specialists to apply advanced machine learning techniques to astronomical data without requiring low-level algorithmic expertise.
• The system is production-ready and open for use by the astronomical community, with demonstrated capability on data sizes comparable to those expected from the LSST.