[Paper Review] Privately Learning Markov Random Fields

Huanyu Zhang, Gautam Kamath+2 more|arXiv (Cornell University)|Feb 21, 2020

Privacy-Preserving Technologies in Data67 references4 citations

TL;DR

This paper studies private learning of Markov Random Fields (MRFs), showing that under differential privacy, structure learning and parameter learning have fundamentally different sample complexity requirements. While non-private MRF learning scales logarithmically with dimension, only approximate differential privacy preserves this efficiency for structure learning; parameter learning and pure/concentrated DP necessitate polynomial sample complexity, revealing a strong privacy-induced separation in high-dimensional settings.

ABSTRACT

We consider the problem of learning Markov Random Fields (including the prototypical example, the Ising model) under the constraint of differential privacy. Our learning goals include both structure learning, where we try to estimate the underlying graph structure of the model, as well as the harder goal of parameter learning, in which we additionally estimate the parameter on each edge. We provide algorithms and lower bounds for both problems under a variety of privacy constraints -- namely pure, concentrated, and approximate differential privacy. While non-privately, both learning goals enjoy roughly the same complexity, we show that this is not the case under differential privacy. In particular, only structure learning under approximate differential privacy maintains the non-private logarithmic dependence on the dimensionality of the data, while a change in either the learning goal or the privacy notion would necessitate a polynomial dependence. As a result, we show that the privacy constraint imposes a strong separation between these two learning problems in the high-dimensional data regime.

Motivation & Objective

• To investigate the sample complexity of learning Markov Random Fields (MRFs) under differential privacy constraints.
• To compare the difficulty of structure learning versus parameter learning in the context of differential privacy.
• To establish tight lower bounds on sample complexity for both learning goals under pure, concentrated, and approximate differential privacy.
• To identify the conditions under which private MRF learning remains efficient in high-dimensional settings.
• To demonstrate that differential privacy fundamentally alters the complexity landscape between structure and parameter learning, unlike in the non-private setting.

Proposed method

• Constructs a family of MRFs with bounded edge weights and structured sparsity to model the underlying graphical model space.
• Uses probabilistic coupling arguments and information-theoretic packing bounds to derive lower bounds on sample complexity under differential privacy.
• Applies the maximal coupling lemma to bound the total variation distance between distributions induced by different MRF parameters.
• Employs the probabilistic packing argument from [ASZ20a] for pure differential privacy and [BS16] for zero-concentrated differential privacy (zCDP).
• Derives lower bounds by analyzing the expected Hamming distance between coupled samples from different MRF distributions.
• Combines results from KL divergence and chi-squared distance to bound privacy loss and derive sample complexity trade-offs.

Experimental results

Research questions

• RQ1 What is the minimum sample complexity required to learn the structure of a pairwise MRF under approximate differential privacy?
• RQ2 How does the sample complexity of parameter learning in MRFs compare to structure learning under pure differential privacy?
• RQ3 Does differential privacy induce a separation in complexity between structure and parameter learning in high-dimensional MRFs?
• RQ4 Can logarithmic sample complexity be preserved for MRF structure learning under concentrated or pure differential privacy?
• RQ5 What are the tightest lower bounds on sample complexity for private MRF learning across different privacy notions?

Key findings

• Under approximate differential privacy, structure learning of pairwise MRFs requires only logarithmic sample complexity in the dimension, matching the non-private case.
• For parameter learning under pure or concentrated differential privacy, the sample complexity grows polynomially in the dimension, even when structure learning remains efficient.
• The paper establishes a lower bound of $ n = \Omega\left(\frac{\sqrt{p}}{\eta\varepsilon} + \frac{k^2 p}{\varepsilon}\right) $ for approximate DP structure learning, where $ p $ is the number of variables, $ \eta $ the minimum edge weight, and $ k $ the alphabet size.
• For zero-concentrated differential privacy, the lower bound is $ n = \Omega\left(\sqrt{\frac{k^2 p}{\rho}}\right) $, showing a polynomial dependence on $ p $.
• The results demonstrate a strong separation between structure and parameter learning under differential privacy: while structure learning remains efficient under approximate DP, parameter learning is significantly more costly across all privacy regimes.
• The study reveals that differential privacy fundamentally changes the complexity landscape of MRF learning, making parameter learning infeasible in high-dimensional settings under strict privacy constraints.