[Paper Review] An adaptive ensemble filter for heavy-tailed distributions: tuning-free inflation and localization

Mathieu Le Provost, Ricardo Baptista+2 more|arXiv (Cornell University)|Oct 12, 2023

Target Tracking and Data Fusion in Sensor NetworksComputer Science3 citations

TL;DR

This paper proposes an adaptive ensemble filter, the EnRF, that enables exact inference for t-distributions in nonlinear state-space models with heavy-tailed noise. By leveraging measure transport and a regularized EM algorithm with l1-penalized covariance estimation, the filter adaptively tunes inflation and localization without manual calibration, outperforming standard EnKF in synthetic heavy-tailed scenarios while recovering Kalman filter performance for Gaussian limits.

ABSTRACT

Heavy tails is a common feature of filtering distributions that results from the nonlinear dynamical and observation processes as well as the uncertainty from physical sensors. In these settings, the Kalman filter and its ensemble version - the ensemble Kalman filter (EnKF) - that have been designed under Gaussian assumptions result in degraded performance. t-distributions are a parametric family of distributions whose tail-heaviness is modulated by a degree of freedom $ν$. Interestingly, Cauchy and Gaussian distributions correspond to the extreme cases of a t-distribution for $ν= 1$ and $ν= \infty$, respectively. Leveraging tools from measure transport (Spantini et al., SIAM Review, 2022), we present a generalization of the EnKF whose prior-to-posterior update leads to exact inference for t-distributions. We demonstrate that this filter is less sensitive to outlying synthetic observations generated by the observation model for small $ν$. Moreover, it recovers the Kalman filter for $ν= \infty$. For nonlinear state-space models with heavy-tailed noise, we propose an algorithm to estimate the prior-to-posterior update from samples of joint forecast distribution of the states and observations. We rely on a regularized expectation-maximization (EM) algorithm to estimate the mean, scale matrix, and degree of freedom of heavy-tailed extit{t}-distributions from limited samples (Finegold and Drton, arXiv preprint, 2014). Leveraging the conditional independence of the joint forecast distribution, we regularize the scale matrix with an $l1$ sparsity-promoting penalization of the log-likelihood at each iteration of the EM algorithm. By sequentially estimating the degree of freedom at each analysis step, our filter can adapt its prior-to-posterior update to the tail-heaviness of the data. We demonstrate the benefits of this new ensemble filter on challenging filtering problems.

Motivation & Objective

• To address degraded performance of the ensemble Kalman filter (EnKF) in systems with heavy-tailed filtering distributions due to nonlinear dynamics and non-Gaussian noise.
• To develop a filtering algorithm that enables exact inference under t-distributed assumptions, which generalize both Gaussian and Cauchy distributions.
• To eliminate the need for manual tuning of inflation and localization parameters by making them data-adaptive through estimation of the t-distribution's degrees of freedom and sparse covariance structure.
• To improve robustness to outliers in synthetic observations by modeling tail-heaviness explicitly via the t-distribution parameter ν.
• To integrate measure transport theory with ensemble filtering to derive a prior-to-posterior transformation that preserves non-Gaussian structure.

Proposed method

• The EnRF uses a transport map-based analysis step that maps forecast samples to posterior samples via a transformation derived from t-distribution theory, enabling exact inference under t-distributed assumptions.
• It estimates the joint mean, scale matrix, and degrees of freedom ν of a t-distribution from limited ensemble samples using a regularized expectation-maximization (EM) algorithm.
• The EM algorithm incorporates l1-penalization on the off-diagonal entries of the inverse scale matrix to promote sparsity, drawing on the graphical lasso framework for robust covariance estimation.
• The filter sequentially estimates ν at each analysis step, allowing the prior-to-posterior update to adapt to the observed tail-heaviness of the data.
• Inflation and localization are intrinsically embedded via the estimated scale matrix and ν: larger ν implies lighter tails and reduced inflation, while sparsity in the precision matrix enforces localization.
• The algorithm is implemented via Cholesky decomposition and linear system solves to efficiently compute posterior samples using a reweighting formula that accounts for the t-distribution's scale and degrees of freedom.

Experimental results

Research questions

• RQ1 Can a filtering algorithm be designed to perform exact inference under t-distributed assumptions in nonlinear, high-dimensional systems with heavy-tailed noise?
• RQ2 How can inflation and localization be made adaptive and data-driven without requiring manual tuning?
• RQ3 To what extent does the proposed EnRF improve robustness to outliers in synthetic observations compared to standard EnKF?
• RQ4 Can the degrees of freedom ν of a t-distribution be reliably estimated from limited ensemble samples using a regularized EM procedure?
• RQ5 Does the proposed filter recover the standard Kalman filter in the limit of ν → ∞, confirming consistency with Gaussian assumptions?

Key findings

• The EnRF demonstrates significantly improved robustness to outlying synthetic observations in heavy-tailed settings, particularly for small ν, where standard EnKF performance degrades.
• The filter recovers the standard Kalman filter in the limit ν → ∞, confirming consistency with Gaussian assumptions.
• The l1-penalized EM algorithm enables stable estimation of sparse precision matrices from limited samples, improving covariance estimation in high-dimensional settings.
• Adaptive inflation emerges naturally from the estimated ν: higher ν leads to lighter tails and reduced inflation, while lower ν increases the influence of extreme observations.
• Localization is implicitly achieved through the sparsity-promoting l1 penalty on the inverse scale matrix, reducing spurious correlations without manual localization tuning.
• Empirical results on challenging filtering problems confirm that the EnRF outperforms standard EnKF in terms of root mean square error and reliability under heavy-tailed noise.