[Paper Review] Uncertainty-Aware Learning for Improvements in Image Quality of the Canada-France-Hawaii Telescope

Sankalp Gilda, Stark C. Draper+7 more|arXiv (Cornell University)|Jun 30, 2021

Advanced Image Processing Techniques48 references7 citations

TL;DR

This paper proposes an uncertainty-aware machine learning framework that predicts image quality (IQ) at the Canada-France-Hawaii Telescope (CFHT) using environmental and operational data. By employing a mixture density network with epistemic and aleatoric uncertainty estimation, the model achieves a mean absolute error of ~0.07′′ in IQ median prediction and identifies optimal dome vent configurations that reduce required observing time by ~12% for fixed signal-to-noise ratio goals.

ABSTRACT

We leverage state-of-the-art machine learning methods and a decade's worth of archival data from CFHT to predict observatory image quality (IQ) from environmental conditions and observatory operating parameters. Specifically, we develop accurate and interpretable models of the complex dependence between data features and observed IQ for CFHT's wide-field camera, MegaCam. Our contributions are several-fold. First, we collect, collate and reprocess several disparate data sets gathered by CFHT scientists. Second, we predict probability distribution functions (PDFs) of IQ and achieve a mean absolute error of $\sim0.07''$ for the predicted medians. Third, we explore the data-driven actuation of the 12 dome "vents" installed in 2013-14 to accelerate the flushing of hot air from the dome. We leverage epistemic and aleatoric uncertainties in conjunction with probabilistic generative modeling to identify candidate vent adjustments that are in-distribution (ID); for the optimal configuration for each ID sample, we predict the reduction in required observing time to achieve a fixed SNR. On average, the reduction is $\sim12\%$. Finally, we rank input features by their Shapley values to identify the most predictive variables for each observation. Our long-term goal is to construct reliable and real-time models that can forecast optimal observatory operating parameters to optimize IQ. We can then feed such forecasts into scheduling protocols and predictive maintenance routines. We anticipate that such approaches will become standard in automating observatory operations and maintenance by the time CFHT's successor, the Maunakea Spectroscopic Explorer, is installed in the next decade.

Motivation & Objective

• To develop accurate, interpretable models of image quality (IQ) at CFHT using archival telemetry and post-processed IQ data.
• To predict full probability distribution functions (PDFs) of IQ from environmental and operational parameters.
• To identify optimal dome vent configurations using uncertainty-aware modeling to improve IQ and reduce observing time.
• To rank input features by their predictive importance using Shapley values for interpretability.
• To lay the foundation for real-time, automated observatory scheduling and maintenance systems.

Proposed method

• Trained a feed-forward mixture density network (MDN) to predict the full probability distribution of IQ from environmental and operational features.
• Integrated epistemic and aleatoric uncertainty estimation to assess model confidence and detect out-of-distribution (OOD) samples.
• Used CRUDE (Calibrated Regression with Uncertainty Estimation) for post-hoc calibration of predictive uncertainty.
• Applied pseudo-marginal log-likelihood thresholds (95th percentile) to detect OOD samples, with log-likelihood regret as a superior but computationally heavier alternative.
• Employed Shapley values to rank feature importance and improve model interpretability.
• Evaluated the impact of dome vent configurations on IQ by simulating in-distribution (ID) adjustments and predicting time savings for fixed SNR.

Experimental results

Research questions

• RQ1 Can machine learning models accurately predict the full probability distribution of image quality at CFHT using environmental and operational data?
• RQ2 What is the optimal configuration of the 12 dome vents that minimizes image degradation and reduces required observing time?
• RQ3 How can epistemic and aleatoric uncertainties be leveraged to identify reliable and actionable control adjustments?
• RQ4 Which environmental and operational features are most predictive of image quality, as measured by Shapley values?
• RQ5 Can uncertainty-aware models enable real-time, data-driven optimization of observatory operations?

Key findings

• The model predicts IQ median with a mean absolute error of approximately 0.07′′, demonstrating high predictive accuracy.
• Optimal dome vent configurations identified via uncertainty-aware modeling reduce required observing time by an average of 12% to achieve a fixed signal-to-noise ratio.
• Shapley value analysis identifies key predictive features, including dome temperature, wind speed, and instrument cooling status, enhancing model interpretability.
• The use of pseudo-marginal log-likelihoods with a 95th percentile threshold effectively detects out-of-distribution samples, though log-likelihood regret is shown to be more accurate but computationally costly.
• The framework enables data-driven actuation of observatory systems, paving the way for automated scheduling and predictive maintenance.
• The study establishes a foundation for real-time, uncertainty-aware IQ forecasting that can be integrated into future observatory control systems, including for the upcoming Maunakea Spectroscopic Explorer.