[Paper Review] The QuickReduce data reduction pipeline for the WIYN One Degree Imager

Ralf Kotulla|arXiv (Cornell University)|Oct 18, 2013

Astronomy and Astrophysical Research3 citations

TL;DR

The QuickReduce pipeline is a fast, Python-based data reduction system for the WIYN One Degree Imager (ODI), enabling real-time quality assessment during observations through parallelized, multi-core processing. It performs full calibration—including crosstalk correction, fringe removal, astrometric solution via 2MASS, and photometric zero-point calibration—achieving sub-minute reduction times per 200 Mpixel frame, and is integrated into both the telescope observer interface and the ODI Portal for cloud-based, on-the-fly processing without raw data downloads.

ABSTRACT

Optimizing one's observing strategy while at the telescope relies on knowing the current observing conditions and the obtained data quality. In particular the latter is not straight forward with current wide-field imagers, such as the WIYN One Degree Imager (ODI), currently consisting of 13 detectors, each of them read out in 64 independent cells. Here we present a fast data reduction software for ODI, optimized for a first data inspection during acquisition at the the telescope, but capable enough for science-quality data reductions. The pipeline is coded in pure python with minimal additional requirements. It is installed on the ODI observer's interface and publicly available from the author's webpage. It performs all basic reduction steps as well as more advanced corrections for pupil-ghost removal, fringe correction and masking of persistent pixels. Additional capabilities include adding an accurate astrometric WCS solution based on the 2MASS reference system as well as photometric zeropoint calibration for frames covered by the SDSS foot-print. The pipeline makes use of multiple CPU-cores wherever possible, resulting in an execution time of only a few seconds per frame. As such this QuickReduce pipeline offers the ODI observer a convenient way to closely monitor data quality, a necessity to optimize the observing strategy during the night.

Motivation & Objective

• To provide real-time data quality monitoring during ODI observations at the WIYN telescope.
• To enable fast, science-quality data reduction with minimal computational and software dependencies.
• To support efficient observing strategy by delivering calibrated, astrometrically and photometrically calibrated frames within seconds.
• To eliminate the need for local data downloads by offering cloud-based processing via the ODI Portal, Pipeline, and Archive (PPA).
• To provide a publicly available, IRAF-free, Python-native pipeline for wide-field imager data reduction.

Proposed method

• The pipeline performs parallel, cell-by-cell reduction across 13 OTAs, combining 64 cells per OTA into single 4K×4K images.
• It applies sequential de-trending: crosstalk correction, overscan subtraction, bias and dark subtraction, non-linearity correction, and flat-fielding.
• Saturated and persistent pixels are masked using catalogs from current and prior frames to correct for charge smear and trails.
• Fringe correction uses static maps and time-variable scaling factors derived from the method of Snodgrass & Carry (2013).
• Astrometric calibration matches source catalogs to 2MASS, achieving rms < 0.2 arcsec, while photometric calibration uses SDSS cross-identification.
• Pupil-ghost removal is achieved via rotator-angle-dependent template generation, eliminating gaps between detectors and cells.

Experimental results

Research questions

• RQ1 How can real-time data quality assessment be achieved for multi-detector wide-field imagers like ODI?
• RQ2 What is the optimal approach to calibrate and reduce 200 Mpixel frames from 13 independent detectors in seconds?
• RQ3 Can a pure Python pipeline with minimal dependencies outperform traditional IRAF-based pipelines in speed and usability?
• RQ4 How can astrometric and photometric solutions be robustly derived for large-format, multi-chip instruments?
• RQ5 To what extent can cloud-based processing eliminate the need for local data downloads in ground-based astronomy?

Key findings

• QuickReduce reduces a 200 Mpixel ODI frame in 20–60 seconds on a 6-core machine with fast I/O, depending on processing steps.
• The pipeline achieves astrometric accuracy of better than 0.2 arcsec rms by matching ODI source catalogs to the 2MASS reference system.
• Photometric zero-point calibration is enabled for frames within the SDSS footprint using cross-identification with SDSS stellar catalogs.
• Fringe correction is effective and adaptive, using time-variable scaling factors derived from a reliable method (Snodgrass & Carry 2013).
• The pipeline is successfully integrated into the ODI Portal, enabling cloud-based, on-the-fly reduction without downloading raw data.
• The code is publicly available and runs without IRAF, relying only on standard Python scientific packages and multiprocessing for parallelization.