[Paper Review] Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables

Jean Daunizeau|arXiv (Cornell University)|Feb 28, 2017

Statistical Methods and Bayesian InferenceMathematics22 citations

TL;DR

This paper presents semi-analytical approximations for the statistical moments of sigmoid and softmax transformations applied to normal random variables. By combining analytical forms with numerically optimized parameters, the method achieves high accuracy—within 5% error—offering a computationally efficient alternative to exact but intractable derivations in models involving probabilistic decision-making and clustering.

ABSTRACT

This note is concerned with accurate and computationally efficient approximations of moments of Gaussian random variables passed through sigmoid or softmax mappings. These approximations are semi-analytical (i.e. they involve the numerical adjustment of parametric forms) and highly accurate (they yield 5% error at most). We also highlight a few niche applications of these approximations, which arise in the context of, e.g., drift-diffusion models of decision making or non-parametric data clustering approaches. We provide these as examples of efficient alternatives to more tedious derivations that would be needed if one was to approach the underlying mathematical issues in a more formal way. We hope that this technical note will be helpful to modellers facing similar mathematical issues, although maybe stemming from different academic prospects.

Motivation & Objective

• To develop accurate and computationally efficient approximations for the statistical moments of sigmoid and softmax functions applied to normally distributed random variables.
• To address the mathematical intractability of exact moment derivations for nonlinear transformations of Gaussian variables in probabilistic models.
• To provide practical tools for researchers in machine learning and neuroscience who require moment estimates without resorting to costly Monte Carlo simulations.
• To demonstrate utility in niche applications such as drift-diffusion models of decision making and non-parametric clustering.
• To offer a technically sound yet accessible alternative to formal derivations that are often tedious and impractical to derive from scratch.

Proposed method

• The approach uses parametric forms derived from analytical approximations of the sigmoid and softmax functions.
• Numerical optimization is applied to fit parameters of these forms to match exact moments computed via Monte Carlo sampling.
• The method leverages known properties of the Gaussian distribution and the nonlinearity of sigmoid/softmax mappings.
• It focuses on approximating first and second-order moments (mean and variance) of transformed variables.
• The resulting approximations are semi-analytical: they are not purely analytical nor fully numerical, but a hybrid that balances accuracy and speed.
• The framework is validated by comparing approximated moments against reference values obtained through high-precision numerical integration or simulation.

Experimental results

Research questions

• RQ1 How can the first and second moments of a sigmoid-transformed normal variable be approximated with high accuracy and low computational cost?
• RQ2 What is the performance of semi-analytical approximations for the softmax transformation of multivariate normal variables?
• RQ3 In what practical modeling contexts do these approximations offer a meaningful advantage over exact or simulation-based methods?
• RQ4 How sensitive are the approximations to the parameters of the underlying normal distribution?
• RQ5 Can these approximations be reliably used in complex models such as drift-diffusion models or non-parametric clustering algorithms?

Key findings

• The proposed semi-analytical approximations achieve a maximum error of less than 5% across a wide range of parameter values.
• The method provides a significant computational advantage over Monte Carlo sampling while maintaining high accuracy.
• The approximations are robust across different means and variances of the input normal variables.
• The approach is directly applicable to drift-diffusion models, where sigmoid transformations model decision thresholds.
• The framework enables efficient moment estimation in non-parametric clustering approaches relying on softmax normalization.
• The numerical fitting procedure ensures that the parametric forms closely match the true moments without requiring full symbolic derivation.