[Paper Review] Generalized Score Matching for Non-Negative Data

Shiqing Yu, Mathias Drton+1 more|PubMed|Dec 26, 2018

Statistical Methods and Inference19 references41 citations

TL;DR

This paper generalizes score matching for non-negative data to improve parameter estimation in exponential-family graphical models with intractable normalizing constants, and develops regularized estimators with theoretical guarantees.

ABSTRACT

A common challenge in estimating parameters of probability density functions is the intractability of the normalizing constant. While in such cases maximum likelihood estimation may be implemented using numerical integration, the approach becomes computationally intensive. The score matching method of Hyvärinen (2005) avoids direct calculation of the normalizing constant and yields closed-form estimates for exponential families of continuous distributions over <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow><mml:msup><mml:mi>R</mml:mi> <mml:mi>m</mml:mi></mml:msup> </mml:mrow> </mml:math> . Hyvärinen (2007) extended the approach to distributions supported on the non-negative orthant, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow><mml:msubsup><mml:mi>R</mml:mi> <mml:mo>+</mml:mo> <mml:mi>m</mml:mi></mml:msubsup> </mml:mrow> </mml:math> . In this paper, we give a generalized form of score matching for non-negative data that improves estimation efficiency. As an example, we consider a general class of pairwise interaction models. Addressing an overlooked inexistence problem, we generalize the regularized score matching method of Lin et al. (2016) and improve its theoretical guarantees for non-negative Gaussian graphical models.

Motivation & Objective

• Motivate the challenge of estimating densities when normalizing constants are intractable.
• Extend Hyvärinen’s non-negative score matching by introducing a generalized h-score matching framework.
• Develop regularized generalized score matching for high-dimensional graphical models.
• Apply the method to pairwise interaction power models on R_+^m and establish theoretical guarantees.
• Demonstrate consistency and practical performance through simulations and RNA-seq data analysis.

Proposed method

• Define generalized h-score matching loss J_h for non-negative data with positive, componentwise functions h_j.
• Prove that J_h is minimized uniquely by P_0 under mild conditions and can be rewritten as an expectation independent of p_0.
• Show that for exponential families, the empirical loss is a quadratic function in the canonical parameter θ, enabling closed-form estimation.
• Introduce regularization by adding a diagonal augmentation to the quadratic form to ensure strong convexity in high dimensions.
• Derive a regularized estimator with an l1 penalty that yields a unique minimizer and analyze its consistency.
• Discuss special cases including univariate truncated normals to illustrate the estimator and its asymptotic properties.

Experimental results

Research questions

• RQ1 How can score matching be generalized to efficiently handle non-negative data with intractable normalizing constants?
• RQ2 What choices of the boundary-dampening functions h_j improve estimation efficiency for non-negative graphical models?
• RQ3 Can regularized generalized score matching yield consistent estimators for high-dimensional non-negative graphical models?
• RQ4 How does the generalized score matching perform on pairwise interaction power models on R_+^m and related truncated GGMs?
• RQ5 What are the theoretical and empirical properties (consistency, asymptotic distribution, robustness) of the proposed estimators?

Key findings

• The generalized h-score matching loss provides a non-negative, boundary-damped objective whose minimizer is uniquely P_0 under mild conditions.
• For exponential families, the empirical loss is quadratic in the canonical parameters, enabling closed-form estimators without computing normalizing constants.
• Adding small diagonal augmentation to the quadratic form yields a bounded, strongly convex loss in high dimensions, preserving consistency under a threshold.
• Regularized generalized score matching with an l1 penalty yields a unique minimizer and supports high-dimensional estimation for non-negative graphical models.
• Special cases illustrate consistent estimation for univariate truncated normals and demonstrate improved efficiency when using bounded or slowly growing h functions.
• The methodology extends to a broad class of pairwise interaction models on R_+^m, including truncated Gaussian graphical models and square-root models, with theoretical guarantees and empirical validation.