[Paper Review] Graphical Models for Non-Negative Data Using Generalized Score Matching

Shiqing Yu, Mathias Drton+1 more|arXiv (Cornell University)|Feb 18, 2018

Advanced Statistical Methods and Models21 references3 citations

TL;DR

This paper proposes a generalized score matching framework for estimating graphical models with non-negative data, improving upon existing methods by replacing the standard $x_j$ weighting with flexible functions $h(x)$ to enhance estimation efficiency. The approach enables consistent, high-dimensional inference for truncated Gaussian graphical models with theoretical guarantees and superior performance in simulations compared to prior methods.

ABSTRACT

A common challenge in estimating parameters of probability density functions is the intractability of the normalizing constant. While in such cases maximum likelihood estimation may be implemented using numerical integration, the approach becomes computationally intensive. In contrast, the score matching method of Hyvärinen (2005) avoids direct calculation of the normalizing constant and yields closed-form estimates for exponential families of continuous distributions over $\mathbb{R}^m$. Hyvärinen (2007) extended the approach to distributions supported on the non-negative orthant $\mathbb{R}_+^m$. In this paper, we give a generalized form of score matching for non-negative data that improves estimation efficiency. We also generalize the regularized score matching method of Lin et al. (2016) for non-negative Gaussian graphical models, with improved theoretical guarantees.

Motivation & Objective

• To address the challenge of intractable normalizing constants in non-negative exponential family distributions.
• To improve estimation efficiency for graphical models on the non-negative orthant by generalizing the score matching loss function.
• To extend regularized score matching to high-dimensional truncated Gaussian graphical models with stronger theoretical guarantees.
• To develop a flexible framework that allows data-adaptive choice of weighting functions $h(x)$ for better performance.

Proposed method

• Proposes a generalized score matching loss that replaces the standard $x_j$ multiplier with a general function $h(x)$, allowing improved boundary handling in non-negative data.
• Derives a loss function that remains quadratic in parameters for exponential families, enabling efficient optimization without computing normalizing constants.
• Applies an $\ell_1$ penalty to the generalized score matching loss to induce sparsity in the inverse covariance matrix $\mathbf{K}$, enabling high-dimensional structure learning.
• Introduces a non-centered parameterization for mean and precision parameters, allowing joint estimation with profiled tuning parameters.
• Uses a profiled estimator for the mean parameter $\bm{\mu}$, with a fixed ratio of tuning parameters $\lambda_{\mathbf{K}} / \lambda_{\bm{\eta}}$ to improve stability and performance.
• Employs mean AUC-preserving averaging for ROC curve construction across multiple trials to evaluate edge recovery performance.

Experimental results

Research questions

• RQ1 Can a generalized score matching framework improve estimation efficiency for non-negative graphical models compared to existing methods?
• RQ2 Does replacing the standard $x_j$ weighting with a flexible function $h(x)$ lead to better finite-sample performance in edge recovery and parameter estimation?
• RQ3 What is the theoretical sample size requirement for consistency of the generalized score matching estimator in high-dimensional truncated Gaussian graphical models?
• RQ4 How does the choice of $h(x)$, including $h(x) = \min(x,3)$, affect estimation accuracy and robustness in finite samples?
• RQ5 Can a profiled estimator with an extra tuning parameter for the mean improve performance over fixed or unpenalized alternatives?

Key findings

• The proposed generalized score matching estimator achieves consistent estimation of the inverse covariance matrix $\mathbf{K}$ in truncated Gaussian graphical models with a sample size requirement of $\Omega(d^2 \log m)$, matching known rates for GGMs.
• All alternative functions $h(x)$ tested, including $h(x) = \min(x,3)$, outperform the standard $h(x) = x^2$ from Hyvärinen (2007) and Lin et al. (2016) in both simulations and theory.
• For $n=1000$ and $m=100$, the profiled estimator with $\lambda_{\bm{\eta}} = 0$ (i.e., no mean penalty) yielded among the worst AUC performance, indicating that tuning the mean parameter significantly improves results.
• Even without tuning the extra parameter, estimators using $h(x) = \min(x,3)$ or other non-quadratic functions performed as well as or better than SPACE and GLASSO in edge recovery.
• ROC curves show that performance improves with increasing $\lambda_{\mathbf{K}} / \lambda_{\bm{\eta}}$ up to a threshold, after which it deteriorates, indicating a trade-off between bias and variance.
• The method maintains a piecewise linear solution path and ensures almost sure uniqueness of the solution under regularity conditions, enabling reliable high-dimensional inference.