[Paper Review] PSD Representations for Effective Probability Models

Alessandro Rudi, Carlo Ciliberto|arXiv (Cornell University)|Jun 30, 2021

Statistical Methods and Inference30 references4 citations

TL;DR

This paper introduces positive semi-definite (PSD) models as a novel framework for probabilistic density estimation that simultaneously enables optimal approximation and learning rates, efficient closed-form computation of sum and product rules via matrix operations, and non-negativity—overcoming the trade-off between expressivity and computational efficiency found in traditional models like mixtures or exponential families.

ABSTRACT

Finding a good way to model probability densities is key to probabilistic inference. An ideal model should be able to concisely approximate any probability while being also compatible with two main operations: multiplications of two models (product rule) and marginalization with respect to a subset of the random variables (sum rule). In this work, we show that a recently proposed class of positive semi-definite (PSD) models for non-negative functions is particularly suited to this end. In particular, we characterize both approximation and generalization capabilities of PSD models, showing that they enjoy strong theoretical guarantees. Moreover, we show that we can perform efficiently both sum and product rule in closed form via matrix operations, enjoying the same versatility of mixture models. Our results open the way to applications of PSD models to density estimation, decision theory and inference.

Motivation & Objective

• To address the long-standing trade-off between expressivity and computational efficiency in probabilistic modeling.
• To develop a class of models that can concisely approximate complex probability densities while supporting exact, efficient operations like marginalization and product rules.
• To establish theoretical guarantees for approximation and generalization in PSD models.
• To provide a compression mechanism to control model complexity without sacrificing accuracy.

Proposed method

• The paper employs a recently proposed class of positive semi-definite (PSD) models that generalize mixture models by allowing negative weights while preserving non-negativity of the resulting function.
• It characterizes the approximation and learning properties of PSD models using smoothness classes of densities, showing minimax optimal rates of order $ n^{-\beta/(2\beta+d)} $ for $ \beta $-times differentiable densities on $ \mathbb{R}^d $.
• It formulates sum and product rules as closed-form matrix operations: marginalization via matrix trace and Schur product, and multiplication via tensor and Schur products.
• It introduces a compression method based on low-rank approximation of the PSD kernel matrix to control the number of components while preserving accuracy.
• The framework is applied to inference tasks such as Hidden Markov Models (HMMs), where all operations (observation, transition, marginalization, normalization) are performed in closed form using matrix algebra.
• Theoretical derivations show that all operations preserve the PSD model structure, enabling recursive inference with consistent parameterization.

Experimental results

Research questions

• RQ1 Can PSD models achieve optimal learning rates for density estimation while maintaining computational efficiency in probabilistic operations?
• RQ2 How do PSD models compare to mixture models and exponential families in terms of approximation power and computational tractability?
• RQ3 Can sum and product rules be performed exactly and efficiently in closed form within the PSD framework?
• RQ4 What is the theoretical relationship between smoothness of the target density and the required number of components in a PSD model?
• RQ5 Can PSD models be effectively used in sequential inference tasks like HMMs without loss of generality or efficiency?

Key findings

• PSD models achieve minimax optimal learning rates of order $ n^{-\beta/(2\beta+d)} $ for $ \beta $-times differentiable densities, overcoming the curse of dimensionality when $ \beta \geq d $.
• The models support exact and efficient sum and product rules via matrix operations, including Schur products and trace operations, enabling closed-form inference.
• PSD models can approximate any non-negative, smooth density with $ m = O(\varepsilon^{-d/\beta}) $ components, achieving concise representation.
• The framework enables recursive Bayesian inference in HMMs with all steps—observation, transition, marginalization, and normalization—computed in closed form using matrix algebra.
• A compression method based on low-rank approximation allows control over model complexity while preserving approximation accuracy and theoretical guarantees.
• The resulting model remains a valid probability density (non-negative everywhere) despite allowing negative weights in the linear combination, resolving a key limitation of traditional mixture models.