[Paper Review] Sparse Graphical Linear Dynamical Systems

Émilie Chouzenoux, V́ıctor Elvira|arXiv (Cornell University)|Jul 6, 2023

Gaussian Processes and Bayesian InferenceComputer Science111 references3 citations

TL;DR

This paper proposes DGLASSO, a novel joint graphical modeling framework for linear-Gaussian state-space models that integrates static graphical Lasso and dynamic Granger-based dependencies. Using a block alternating majorization-minimization algorithm, DGLASSO achieves superior parameter estimation and sparsity recovery on synthetic and real weather data, outperforming existing methods in accuracy and convergence.

ABSTRACT

Time-series datasets are central in machine learning with applications in numerous fields of science and engineering, such as biomedicine, Earth observation, and network analysis. Extensive research exists on state-space models (SSMs), which are powerful mathematical tools that allow for probabilistic and interpretable learning on time series. Learning the model parameters in SSMs is arguably one of the most complicated tasks, and the inclusion of prior knowledge is known to both ease the interpretation but also to complicate the inferential tasks. Very recent works have attempted to incorporate a graphical perspective on some of those model parameters, but they present notable limitations that this work addresses. More generally, existing graphical modeling tools are designed to incorporate either static information, focusing on statistical dependencies among independent random variables (e.g., graphical Lasso approach), or dynamic information, emphasizing causal relationships among time series samples (e.g., graphical Granger approaches). However, there are no joint approaches combining static and dynamic graphical modeling within the context of SSMs. This work proposes a novel approach to fill this gap by introducing a joint graphical modeling framework that bridges the graphical Lasso model and a causal-based graphical approach for the linear-Gaussian SSM. We present DGLASSO (Dynamic Graphical Lasso), a new inference method within this framework that implements an efficient block alternating majorization-minimization algorithm. The algorithm's convergence is established by departing from modern tools from nonlinear analysis. Experimental validation on various synthetic data showcases the effectiveness of the proposed model and inference algorithm.

Motivation & Objective

• To address the gap in existing methods that treat static and dynamic dependencies separately in state-space models (SSMs).
• To develop a unified framework that combines static graphical Lasso and dynamic Granger-based modeling within linear-Gaussian SSMs.
• To enable sparse estimation of model parameters—particularly the transition matrix A—while incorporating both statistical and causal dependencies.
• To design an efficient inference algorithm with strong convergence guarantees for complex SSM parameter estimation.
• To validate the method on synthetic data and real-world weather variability datasets, demonstrating improved performance over state-of-the-art approaches.

Proposed method

• Proposes a joint graphical modeling framework that integrates static (graphical Lasso) and dynamic (Granger-based) dependencies in linear-Gaussian SSMs.
• Introduces DGLASSO, an inference algorithm based on block alternating majorization-minimization to optimize the likelihood with sparsity-inducing penalties.
• Uses proximal algorithms to handle non-smooth sparsity-inducing terms in the optimization, enabling efficient computation.
• Employs an EM-like structure where the E-step computes posterior means and covariances via the Kalman filter and RTS smoother.
• Applies majorization techniques to construct a surrogate function that upper-bounds the objective, ensuring monotonic convergence.
• Establishes convergence using tools from nonlinear analysis, including Kurdyka–Łojasiewicz inequality, ensuring global convergence under mild conditions.

Figure 4 : Evolution of RMSE, F1, cNMSE and loss scores on the estimation of ${\mathbf{A}}$ (left) and ${\mathbf{P}}$ (right) by DGLASSO, as a function of hyperparameters $(\lambda_{A},\lambda_{P})$ , for dataset A (averaged on 10 runs). As a comparison, the averaged RMSE scores for $(\lambda_{A},\l

Experimental results

Research questions

• RQ1 Can a unified framework jointly model static and dynamic dependencies in linear-Gaussian state-space models?
• RQ2 How can sparsity be effectively imposed on the transition matrix A while preserving interpretability and estimation accuracy?
• RQ3 Does the proposed DGLASSO algorithm achieve better parameter estimation and graph recovery than existing methods like GLASSO, rGLASSO, MLEM, and GRAPHEM?
• RQ4 What is the convergence behavior of the proposed block alternating majorization-minimization algorithm in high-dimensional SSMs?
• RQ5 How does the method perform on real-world time-series data with complex temporal and cross-variable dependencies?

Key findings

• DGLASSO achieves the lowest RMSE for estimating the transition matrix A across all sparsity levels, with RMSE = 0.0605 at s_A = 27, outperforming MLEM (0.0761) and GRAPHEM (0.0446) in this setting.
• At s_A = 10, DGLASSO achieves an AUC of 0.9814 for graph recovery, significantly outperforming rGLASSO (0.7494) and GLASSO (0.6667).
• For state distribution estimation, DGLASSO achieves cNMSE = 9.3856×10⁻⁷ at s_A = 10, outperforming MLEM (2.3831×10⁻⁶) and GRAPHEM (8.9556×10⁻⁵).
• The predictive distribution likelihood (log-likelihood) is minimized by DGLASSO at 12,290.79 for s_A = 10, compared to 29,034.72 for GRAPHEM, indicating superior predictive accuracy.
• DGLASSO maintains stable convergence and consistently outperforms MLEM, GLASSO, rGLASSO, and GRAPHEM across all sparsity levels in AUC, F1, and RMSE metrics.
• The algorithm demonstrates strong convergence properties, validated via modern nonlinear analysis tools, including the Kurdyka–Łojasiewicz inequality.

Figure 5 : Box plots for the RMSE and F1 scores when running MLEM (left) and DGLASSO (right) on 50 randomly generated LG-SSM time series, for dataset A (top) and dataset D (bottom). DGLASSO outperforms MLEM on all runs, in terms of accuracy metrics (the higher, the better), while its RMSE scores are