[Paper Review] Concrete Score Matching: Generalized Score Matching for Discrete Data

Chenlin Meng, Kristy Choi+2 more|arXiv (Cornell University)|Nov 2, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper introduces Concrete Score Matching (CSM), a novel framework that generalizes score matching to discrete data by defining a new 'Concrete score' based on local directional changes in a predefined neighborhood structure. By using Manhattan distance to measure changes, CSM enables gradient-like optimization in discrete spaces, achieving state-of-the-art performance in density estimation across synthetic, tabular, and high-dimensional image datasets.

ABSTRACT

Representing probability distributions by the gradient of their density functions has proven effective in modeling a wide range of continuous data modalities. However, this representation is not applicable in discrete domains where the gradient is undefined. To this end, we propose an analogous score function called the "Concrete score", a generalization of the (Stein) score for discrete settings. Given a predefined neighborhood structure, the Concrete score of any input is defined by the rate of change of the probabilities with respect to local directional changes of the input. This formulation allows us to recover the (Stein) score in continuous domains when measuring such changes by the Euclidean distance, while using the Manhattan distance leads to our novel score function in discrete domains. Finally, we introduce a new framework to learn such scores from samples called Concrete Score Matching (CSM), and propose an efficient training objective to scale our approach to high dimensions. Empirically, we demonstrate the efficacy of CSM on density estimation tasks on a mixture of synthetic, tabular, and high-dimensional image datasets, and demonstrate that it performs favorably relative to existing baselines for modeling discrete data.

Motivation & Objective

• To address the limitation of traditional score matching, which relies on gradients and fails in discrete domains where gradients are undefined.
• To define a meaningful analog of the (Stein) score for discrete data by leveraging structural neighborhood information.
• To develop a scalable training objective for learning discrete score functions from samples, enabling application to high-dimensional discrete data.
• To demonstrate that the proposed Concrete Score Matching framework outperforms existing baselines in density estimation tasks on diverse discrete data modalities.

Proposed method

• Proposes the Concrete score as a generalization of the (Stein) score, defined as the rate of change of probabilities with respect to local directional changes in a discrete space.
• Uses Manhattan distance to measure directional changes, yielding a novel score function suitable for discrete domains, while recovering the continuous (Stein) score under Euclidean distance.
• Introduces Concrete Score Matching (CSM), a framework to learn the Concrete score from data samples using a tractable training objective.
• Develops an efficient learning objective that scales to high-dimensional data by leveraging conditional probabilities and neighborhood structures.
• Adapts techniques from continuous score matching, such as denoising score matching, to the discrete setting via the Concrete score formulation.
• Employs a reparameterization trick and marginalization-based simplifications to derive a stable and differentiable training objective.

Experimental results

Research questions

• RQ1 Can a surrogate gradient-like function be defined for discrete probability distributions where the true gradient is undefined?
• RQ2 How can the concept of the (Stein) score be generalized to discrete domains using structural neighborhood information?
• RQ3 Can a scalable and stable training objective be derived for learning such generalized scores from data samples?
• RQ4 Does the proposed Concrete Score Matching framework achieve competitive performance in density estimation on discrete data compared to existing methods?
• RQ5 Can the framework be successfully extended to high-dimensional discrete data, such as images and tabular data?

Key findings

• The Concrete score successfully generalizes the (Stein) score to discrete domains by measuring local probability changes via Manhattan-distance-based directional perturbations.
• The proposed Concrete Score Matching (CSM) framework achieves favorable performance on density estimation tasks across synthetic, tabular, and high-dimensional image datasets.
• Models trained with the corrected objective (Equation 32) produce more reasonable samples than those trained with the original typo-ridden version (Equation 31).
• The method scales effectively to high-dimensional discrete data, demonstrating the feasibility of applying score-based generative modeling to structured discrete distributions.
• Empirical results show that CSM outperforms existing baselines in discrete density estimation, particularly in capturing complex dependencies in tabular and image data.
• The framework establishes a principled connection between continuous and discrete score matching, enabling the transfer of recent advances in continuous score-based models to discrete settings.