[Paper Review] A New Forced Photometry Service for the Zwicky Transient Facility

Frank J. Masci, Russ R. Laher+7 more|arXiv (Cornell University)|May 25, 2023

Spectroscopy and Laser Applications11 citations

TL;DR

The paper describes the ZTF Forced Photometry Service (ZFPS), its batch-enabled workflow, architecture, and photometry methodology to generate publication-quality lightcurves from difference images.

ABSTRACT

We describe the Zwicky Transient Facility (ZTF) Forced Photometry Service (ZFPS) as developed and maintained by the ZTF Science Data System Team at IPAC/Caltech. The service is open for public use following a subscription. The ZFPS has been operational since early 2020 and has been used to generate publication quality lightcurves for a myriad of science programs. The ZFPS has been recently upgraded to allow users to request forced-photometry lightcurves for up to 1500 sky positions per request in a single web-application submission. The underlying software has been recoded to take advantage of a parallel processing architecture with the most compute-intensive component rewritten in C and optimized for the available hardware. The ZTF processing cluster consists of 66 compute nodes, each hosting at least 16 physical cores. The compute nodes are generally idle following nightly real-time processing of the ZTF survey data and when other ad hoc processing tasks have been completed. The ZFPS and associated infrastructure at IPAC/Caltech therefore enable thousands of forced-photometry lightcurves to be generated along with a wealth of quality metrics to facilitate analyses and filtering of bad quality data prior to scientific use.

Motivation & Objective

• Motivate the need for forced photometry to recover measurements below single-epoch detection thresholds in ZTF data.
• Describe the design, deployment, and operation of the ZFPS as a batch-processing service.
• Detail the photometry methodology, performance considerations, and data-quality metrics used to produce lightcurves.
• Provide user guidance for submitting requests, retrieving results, and constructing publication-quality lightcurves.

Proposed method

• Batch-submission workflow enabling up to 1500 sky positions per request.
• C-based, multi-threaded PSF-fit photometry plus aperture photometry on difference-image stamps.
• Upsampling of PSFs and stamps by a factor of five for precise centroiding and flux extraction.
• Quality flags, baseline corrections, and uncertainty validation to ensure reliable lightcurves.
• Conversion of differential fluxes to calibrated magnitudes with baseline handling and upper-limit computation.

Figure 1: (a) Number of forced photometry requests submitted versus calendar date since January 1, 2021. (b) Number of forced photometry lightcurves generated as a function of lightcurve time-span, also from requests submitted since January 1, 2021.

Experimental results

Research questions

• RQ1 How can forced photometry be scaled to handle large numbers of sky positions per submission in ZTF data?
• RQ2 What are the performance, memory, and compute requirements to process forced-photometry lightcurves at scale on the ZTF cluster?
• RQ3 How should differential flux measurements from difference images be processed, validated, and converted to calibrated magnitudes for scientific use?
• RQ4 What quality metrics and baseline corrections are necessary to produce reliable lightcurves from ZFPS?
• RQ5 What are practical operational guidelines for users submitting, monitoring, and retrieving ZFPS results?

Key findings

• The ZFPS now supports batch processing of up to 1500 sky positions per submission, a significant throughput improvement over the initial version.
• The service runs on a 66-node cluster with at least 16 cores per node, leveraging parallel processing and a C core for PSF-fitting photometry.
• Photometry is performed on 25x25 pixel difference-image stamps, upsampled to 125x125, with PSF-fit flux, flux uncertainty, and aperture photometry included in outputs.
• Quality filtering, baseline correction, and uncertainty validation are emphasized to produce credible lightcurves, including handling non-detections via upper limits.
• The lightcurves include per-epoch status codes and metadata, enabling robust filtering and analysis for publication-quality use.

Figure 2: ZFPS data flow and system architecture. Major components consist of a compute cluster managed by SLURM; an operations database for retrieving image inputs and metadata; a SQLite database for storage of ZFPS products and a web-server for serving them to users.