[Paper Review] Sparse and Smooth Prior for Bayesian Linear Regression with Application to ETEX Data

Lukáš Ulrych, Václav Šmídl|arXiv (Cornell University)|Jun 21, 2017

Statistical Methods and Inference21 references3 citations

TL;DR

This paper proposes a novel Bayesian linear regression model with a sparse and smooth prior using a least-squares adaptive prior covariance (LS-APC) framework, enabling flexible, data-driven trade-offs between sparsity and smoothness. The method outperforms Bayesian Fused Lasso in estimating complex, non-piecewise-constant release profiles and provides more conservative uncertainty bounds in low-sensitivity regions, as validated on ETEX atmospheric tracer data.

ABSTRACT

Sparsity of the solution of a linear regression model is a common requirement, and many prior distributions have been designed for this purpose. A combination of the sparsity requirement with smoothness of the solution is also common in application, however, with considerably fewer existing prior models. In this paper, we compare two prior structures, the Bayesian fused lasso (BFL) and least-squares with adaptive prior covariance matrix (LS-APC). Since only variational solution was published for the latter, we derive a Gibbs sampling algorithm for its inference and Bayesian model selection. The method is designed for high dimensional problems, therefore, we discuss numerical issues associated with evaluation of the posterior. In simulation, we show that the LS-APC prior achieves results comparable to that of the Bayesian Fused Lasso for piecewise constant parameter and outperforms the BFL for parameters of more general shapes. Another advantage of the LS-APC priors is revealed in real application to estimation of the release profile of the European Tracer Experiment (ETEX). Specifically, the LS-APC model provides more conservative uncertainty bounds when the regressor matrix is not informative.

Motivation & Objective

• To address the lack of Bayesian prior models that jointly enforce sparsity and smoothness in high-dimensional linear regression.
• To develop a Gibbs sampling inference algorithm for the LS-APC model, previously only solved via variational Bayes.
• To compare LS-APC with Bayesian Fused Lasso (BFL) in both simulation and real atmospheric inverse modeling using ETEX data.
• To evaluate the robustness and uncertainty quantification of LS-APC under low-sensitivity measurement conditions.
• To enable model selection among different measurement covariance structures using Bayesian marginal likelihood

Proposed method

• Proposes a hierarchical prior structure where regression coefficients are conditionally normal with precision hyperpriors, and introduces a latent correlation parameter $ l_i $ to control the trade-off between sparsity and smoothness.
• Models the prior covariance as $ \beta_i \sim \mathcal{N}(-l_i \beta_{i+1}, \tau_i^{-1}) $, with $ l_i \sim \mathcal{N}(l_0, \psi_i^{-1}) $, allowing adaptive shrinkage toward smooth or sparse solutions.
• Derives a full Gibbs sampling algorithm for posterior inference and Bayesian model selection, enabling accurate uncertainty quantification.
• Compares the Gibbs sampling (GS) and variational Bayes (VB) approximations of the LS-APC model in terms of accuracy and uncertainty calibration.
• Applies the model to the ETEX atmospheric tracer experiment, using both full and reduced data sets to assess performance under varying data informativeness.
• Uses the marginal likelihood (via variational lower bound) for model selection across different measurement error covariance structures

Experimental results

Research questions

• RQ1 How does the LS-APC prior perform compared to the Bayesian Fused Lasso in estimating non-piecewise-constant release profiles in atmospheric inverse modeling?
• RQ2 Can a Gibbs sampling algorithm be effectively derived for the LS-APC model to improve uncertainty quantification over existing variational Bayes approaches?
• RQ3 How do the uncertainty bounds of LS-APC compare to BFL in low-sensitivity time periods where data are uninformative?
• RQ4 Does the LS-APC model provide more conservative and reliable uncertainty estimates than BFL in real-world atmospheric release estimation?
• RQ5 How well does the variational Bayes approximation of LS-APC align with the Gibbs sampling results in terms of model selection and marginal likelihood estimation?

Key findings

• The LS-APC model with Gibbs sampling (LS-APC-GS) achieved lower absolute error (181.49) on the full ETEX data set compared to Bayesian Fused Lasso (233.80), indicating superior estimation accuracy.
• On ETEX 66 data, LS-APC-GS achieved an absolute error of 186.52, outperforming BFL (215.98), especially in regions with low measurement sensitivity.
• The LS-APC-GS method produced significantly wider 95% credible intervals in uninformative time periods (e.g., before index 55), reflecting more conservative uncertainty quantification than VB or BFL.
• The variational Bayes (VB) approximation of LS-APC provided narrower uncertainty bounds than GS, but the VB lower bound for marginal likelihood closely matched the GS-based estimate, validating its use for model selection.
• In the absence of smoothness (i.e., $ l = 0 $), LS-APC with only sparsity prior produced higher errors (231.19 on ETEX), confirming that smoothness improves estimation when data are informative.
• The LS-APC model with adaptive $ l_i $ estimation successfully captured complex release shapes, outperforming the piecewise-constant assumption of BFL in real data