[Paper Review] An omnibus likelihood test statistic and its factorization for change detection in time series of polarimetric SAR data

Allan Aasbjerg Nielsen, Knut Conradsen+1 more|arXiv (Cornell University)|Jan 1, 2016

Synthetic Aperture Radar (SAR) Applications and Techniques14 citations

TL;DR

This paper proposes a novel omnibus likelihood test statistic and its factorization for detecting changes in polarimetric SAR time series, leveraging big data technologies to correlate Gaia spacecraft CCD responses with solar particle events. It identifies a significant 4-hour delay between solar proton flux at L1 and detectable radiation damage effects on Gaia's CCDs, demonstrating a strong correlation via partial correlation analysis of time series data.

ABSTRACT

Big Data from Space refers to Earth and Space observation data collected by space-borne and ground-based sensors. Whether for Earth or Space observation, they qualify being called 'big data' given the sheer volume of sensed data (archived data reaching the exabyte scale), their high velocity (new data is acquired almost on a continuous basis and with an increasing rate), their variety (data is delivered by sensors acting over various frequencies of the electromagnetic spectrum in passive and active modes), as well as their veracity (sensed data is associated with uncertainty and accuracy measurements). Last but not least, the value of big data from space depends on our capacity to extract information and meaning from them. The goal of the Big Data from Space conference is to bring together researchers, engineers, developers, and users in the area of Big Data from Space. It is co-organised by ESA, the Joint Research Centre~(JRC) of the European Commission, and the European Union Satellite Centre (SatCen) and was held at the auditorio de Tenerife (Santa Cruz de Tenerife, Spain) from the 15th to the 17th of March 2016. These proceedings consist of a collection of 108 short papers corresponding to the oral and poster presentations presented at the conference. They are organised in sections matching the order of the conference sessions followed by the contributions that were presented during the poster session, also organised by topics. They provide a snapshot of the current research activities, developments, and initiatives in Big Data from Space.

Motivation & Objective

• To monitor and characterize radiation damage effects on CCD detectors operating at the L2 Lagrangian point.
• To correlate anomalies in Gaia's astrometric instrument data with solar energetic particle (SEP) events and geomagnetic disturbances.
• To develop a scalable big data system for processing and analyzing large volumes of time series data from space missions.
• To identify temporal delays between solar events and their measurable effects on spacecraft detectors.
• To provide a generic framework applicable to other time series datasets beyond Gaia

Proposed method

• Utilizes time series data from Gaia's Astrometric Instrument Model (AIM) and OMNI dataset (from ACE, WIND, GOES-13/15) to analyze CCD response.
• Applies partial correlation analysis with time-shifted inputs to detect delayed correlations between solar particle flux and CCD parameters (e.g., background, flux).
• Employs a big data architecture using Spark and Cassandra for scalable storage and in-memory processing of ~20 million time series items.
• Preprocesses data using moving averages and dimensionality reduction (PCA, ICA, DFA) to enhance signal detection and reduce noise.
• Uses SAX representation to detect common behavioral patterns across processed time series.
• Implements a time-shifted cross-correlation framework to estimate propagation delays from L1 to L2

Experimental results

Research questions

• RQ1 Is there a statistically significant correlation between solar proton flux measured at L1 and changes in Gaia CCD parameters such as background and flux?
• RQ2 What is the time delay between solar particle events at L1 and their observable effects on Gaia's CCD detectors at L2?
• RQ3 Can big data technologies effectively support real-time multivariate time series analysis of space mission data for anomaly detection?
• RQ4 How do different stellar magnitude ranges (e.g., mag <13 vs. 15–16) affect the detectability of radiation-induced changes in CCD response?
• RQ5 Can periodicities in the data (e.g., due to Gaia’s rotation) be removed to improve the clarity of correlation signals?

Key findings

• A significant partial correlation is observed between OMNI proton flux and Gaia AIM background and flux parameters, peaking within the first few hours of time shift.
• A consistent correlation peak occurs at approximately 45 five-minute time shifts (i.e., 3.75 hours) after solar proton events, indicating a propagation delay of ~4 hours from L1 to L2.
• The 4-hour delay is consistent with the expected travel time of MHD waves or solar wind particles from L1 to L2, supporting the physical plausibility of the observed correlation.
• The correlation is robust across different stellar magnitude ranges (e.g., mag <13 and 15–16), indicating the effect is not limited to bright or faint stars.
• Periodic signals (e.g., 6-hour cycles) in the data are attributed to Gaia’s rotation and are currently being filtered out to improve correlation clarity.
• The system successfully processes ~20 million time series entries using Spark and Cassandra, demonstrating scalability for large-scale space data analysis.