[Paper Review] Automatic Reparameterisation of Probabilistic Programs

Maria I. Gorinova, Dave Moore+1 more|arXiv (Cornell University)|Jun 7, 2019

Machine Learning and Algorithms23 references4 citations

TL;DR

This paper proposes automatic reparameterization in probabilistic programming to improve inference robustness and efficiency by dynamically switching between centred and non-centred parameterizations. It introduces two algorithms—interleaved HMC and Variationally Inferred Parameterisation (VIP)—that outperform fixed parameterizations, with VIP achieving up to 3× higher effective sample size per gradient evaluation on challenging models like German credit.

ABSTRACT

Probabilistic programming has emerged as a powerful paradigm in statistics, applied science, and machine learning: by decoupling modelling from inference, it promises to allow modellers to directly reason about the processes generating data. However, the performance of inference algorithms can be dramatically affected by the parameterisation used to express a model, requiring users to transform their programs in non-intuitive ways. We argue for automating these transformations, and demonstrate that mechanisms available in recent modeling frameworks can implement non-centring and related reparameterisations. This enables new inference algorithms, and we propose two: a simple approach using interleaved sampling and a novel variational formulation that searches over a continuous space of parameterisations. We show that these approaches enable robust inference across a range of models, and can yield more efficient samplers than the best fixed parameterisation.

Motivation & Objective

• To address the challenge of poor inference performance due to suboptimal model parameterization in probabilistic programming.
• To automate non-centring and related reparameterizations, reducing the need for manual, non-intuitive model rewriting.
• To enable inference algorithms that dynamically select or learn optimal parameterizations without prior knowledge of the best form.
• To decouple model specification from inference efficiency, allowing users to focus on statistical assumptions.

Proposed method

• Leverages effect-handling mechanisms in modern probabilistic programming frameworks to automatically apply bijective transformations (e.g., non-centring) to latent variables.
• Proposes interleaved Hamiltonian Monte Carlo (iHMC), which alternates HMC steps between centred and non-centred parameterizations to improve mixing.
• Introduces Variationally Inferred Parameterisation (VIP), a novel variational inference method that searches over a continuous space of parameterizations including non-centring as a special case.
• Uses a mean-field variational approximation to jointly optimize model parameters and reparameterization parameters (λ), with λ controlling the degree of non-centring.
• Employs gradient-based optimization to tune parameterization hyperparameters, enabling automatic discovery of optimal configurations.
• Applies diagonal preconditioning using variational posteriors to improve HMC efficiency across all methods.

Experimental results

Research questions

• RQ1 Can automatic reparameterization improve the robustness and efficiency of inference across diverse probabilistic models?
• RQ2 Can inference algorithms dynamically discover better parameterizations than fixed human-designed ones?
• RQ3 Does a continuous search over parameterizations (via VIP) yield better performance than discrete choices like centred or non-centred forms?
• RQ4 Can the ELBO serve as a reliable proxy for posterior geometry and sampling efficiency in reparameterized models?
• RQ5 How does automatic reparameterization compare to manual non-centring in terms of sampling efficiency and convergence?

Key findings

• iHMC achieved at least as good performance as the best fixed parameterization (CP-HMC or NCP-HMC) across all datasets, and outperformed both on the German credit dataset with 3.0±0.2 ESS/∇.
• VIP-HMC achieved 5.6±0.6 ESS/∇ on the German credit dataset—over three times higher than CP-HMC and NCP-HMC—demonstrating superior sampling efficiency.
• VIP found meaningful, interpretable parameterizations: most variables were set to either centred (λ=0) or non-centred (λ=1), but mixed configurations emerged in complex cases like German credit.
• The ELBO was found to correlate well with sampling efficiency, validating its use as a predictor of model conditioning and a guide for automatic reparameterization.
• On average, VIP-HMC matched or exceeded the performance of the best fixed parameterization, confirming that automated search can discover superior configurations.