[Paper Review] Bayesian Properties of Normalized Maximum Likelihood and its Fast Computation

Andrew R. Barron, Teemu Roos+1 more|arXiv (Cornell University)|Jan 28, 2014

Gaussian Processes and Bayesian Inference5 references4 citations

TL;DR

This paper establishes an exact Bayes-like representation of the normalized maximum likelihood (NML) distribution as a finite mixture of component models, even with signed (positive and negative) prior weights, under mild linear independence conditions. The method enables fast and exact computation of NML marginals and conditionals via linear algebra or optimization, significantly improving efficiency in universal coding and prediction tasks.

ABSTRACT

The normalized maximized likelihood (NML) provides the minimax regret solution in universal data compression, gambling, and prediction, and it plays an essential role in the minimum description length (MDL) method of statistical modeling and estimation. Here we show that the normalized maximum likelihood has a Bayes-like representation as a mixture of the component models, even in finite samples, though the weights of linear combination may be both positive and negative. This representation addresses in part the relationship between MDL and Bayes modeling. This representation has the advantage of speeding the calculation of marginals and conditionals required for coding and prediction applications.

Motivation & Objective

• To establish an exact representation of the normalized maximum likelihood (NML) as a Bayes-like mixture, even with signed prior weights, in finite samples.
• To resolve the theoretical relationship between minimum description length (MDL) and Bayesian modeling by showing that signed priors are necessary for exact NML representation in some models.
• To develop computationally efficient methods for calculating NML marginals and conditionals required in data compression and prediction.
• To demonstrate that the number of mass points required for exact representation is bounded by the model's degrees of freedom, particularly for discrete exponential families.
• To explore the implications of this representation for predictive performance, including matching posterior means to maximum likelihood estimates in Bernoulli trials.

Proposed method

• Formulate NML as a finite mixture of component densities $ p(x| heta_k) $, with weights $ W_k $ that may be positive or negative, using linear algebra to solve for weights given fixed $ \theta_k $.
• Use a Newton-type optimization algorithm to jointly estimate both the locations $ \theta_k $ and weights $ W_k $ of the mass points, minimizing a Renyi divergence criterion with large $ \beta $ to approximate worst-case ratio.
• Apply the optimization under the constraint $ \sum_k W_k = 1 $, with $ \beta = 150 $ (or $ 120 $ for $ n=500 $) to ensure convergence to the minimax regret solution.
• Initialize mass points at $ \sin^2((k-1)\pi / 2(K-1)) $ to improve convergence and avoid degeneracy.
• Verify accuracy via Kullback-Leibler divergence $ D(\text{NML} \| \text{mix}) < 10^{-7} $ and worst-case ratio $ \max_x \text{NML}(x)/\text{mix}(x) < 1 + 10^{-3} $.
• Demonstrate the method on Bernoulli models with $ n $ i.i.d. trials, showing that $ \lfloor n/2 \rfloor + 1 $ or $ n+1 $ mass points suffice for exact representation.

Experimental results

Research questions

• RQ1 Can the normalized maximum likelihood (NML) be exactly represented as a finite Bayes mixture, even when prior weights are allowed to be negative?
• RQ2 What is the minimal number of mass points required to exactly represent NML in a given parametric model, and how does this relate to the model's degrees of freedom?
• RQ3 How can the NML distribution be efficiently computed for marginal and conditional probability estimation in universal coding and prediction?
• RQ4 Under what conditions does a Bayes-like representation of NML require signed (positive and negative) prior weights rather than non-negative ones?
• RQ5 Can the NML matching prior be constructed such that the posterior mean of the log odds matches the maximum likelihood estimate in Bernoulli trials?

Key findings

• An exact representation of NML as a finite mixture of component models exists, even with signed prior weights, provided a mild linear independence condition on the model is satisfied.
• For the Bernoulli model with $ n $ i.i.d. trials, $ \lfloor n/2 \rfloor + 1 $ mass points suffice for exact NML representation when both $ \theta_k $ and $ W_k $ are optimized.
• With $ n+1 $ mass points, an exact representation is guaranteed regardless of $ \theta_k $, and optimization of $ \theta_k $ further improves approximation accuracy.
• The proposed optimization method achieves $ D(\text{NML} \| \text{mix}) < 10^{-7} $ and worst-case ratio $ \max_x \text{NML}(x)/\text{mix}(x) < 1 + 10^{-3} $, confirming high fidelity.
• The method enables fast and exact computation of NML marginals and conditionals, which are essential for arithmetic coding and sequential prediction.
• In Bernoulli trials, the NML-matching prior ensures the posterior mean of the log odds equals the MLE of the log odds whenever the number of ones is neither 0 nor $ n $.