[Paper Review] The knee-jerk mapping

Peter G. Doyle, Jim Reeds|ArXiv.org|Jun 2, 2006

Machine Learning and Algorithms6 references3 citations

TL;DR

This paper presents a rigorous theoretical foundation for the 'knee-jerk mapping,' a key component in optimization algorithms like the EM algorithm. It shows that the mapping, defined by scaling variables by their partial derivatives, monotonically increases the objective function for log-log-convex functions on the simplex, due to convexity of log Z in log-space and maximization of a lower bound via I-divergence minimization.

ABSTRACT

We claim to give the definitive theory of what we call the `knee-jerk mapping', which is the basis for a class of optimization algorithms introduced by Baum, and promoted by Dempster, Laird, and Rubin under the name `EM algorithm'.

Motivation & Objective

• To establish a definitive theoretical basis for the knee-jerk mapping used in optimization algorithms such as EM.
• To explain why the knee-jerk mapping strictly increases the value of a log-log-convex function on the simplex.
• To unify the behavior of the mapping under convexity and divergence minimization principles.
• To generalize the mapping to weighted simplices and products of simplices.
• To demonstrate that the mapping's convergence is guaranteed by the structure of log-log-convex and log-concave functions.

Proposed method

• Define the knee-jerk mapping as $ x' = T_Z(x) = \frac{1}{\sum_i x_i Z_{x_i}} (x_1 Z_{x_1}, \dots, x_n Z_{x_n}) $, which maps the positive orthant to the simplex.
• Use the characterization that $ Z $ is log-log-convex iff $ \log Z $ is convex in $ (\log x_1, \dots, \log x_n) $, ensuring the tangent plane inequality holds.
• Maximize the lower bound of $ \log Z(\bar{x}) - \log Z(x) $ using Lagrange multipliers under the simplex constraint $ \sum x_i = 1 $, leading to the knee-jerk update.
• Apply Jensen’s inequality to show that the I-divergence $ I({\bf x}'; {\bf x}) \geq 0 $, which underpins the monotonic increase in $ Z $.
• Generalize the mapping to weighted simplices $ \Sigma_{{\bf a}} $ via $ x'_i = \frac{x_i Z_{x_i} / a_i}{\sum_j x_j Z_{x_j} / a_j} $, preserving monotonicity.
• Extend the framework to products of simplices, where the mapping acts independently per simplex component, maintaining the inequality via sum of I-divergences.

Experimental results

Research questions

• RQ1 Why does the knee-jerk mapping strictly increase the value of a log-log-convex function on the simplex?
• RQ2 What is the geometric and information-theoretic justification for the mapping’s monotonicity?
• RQ3 How does the mapping relate to the EM algorithm and other optimization schemes?
• RQ4 Can the mapping be generalized to non-standard simplex constraints, such as weighted or product simplices?
• RQ5 What role does log-concavity play in ensuring uniqueness of the maximum for such functions?

Key findings

• The knee-jerk mapping increases $ Z $ at each iteration because it maximizes a lower bound derived from the convexity of $ \log Z $ in log-space.
• The inequality $ \log \frac{Z'}{Z} \geq \frac{\sum x_i Z_{x_i}}{Z} I({\bf x}'; {\bf x}) \geq 0 $ proves monotonic improvement, with equality iff $ {\bf x} $ is a critical point.
• The function $ x_1 + \cdots + x_n $ is a fundamental knee-jerk function, and its log-log-convexity underpins the I-divergence non-negativity via Jensen’s inequality.
• Discriminants of graphs (e.g., $ D_G = xy + xz + yz $) are log-concave and log-log-convex, making them valid candidates for knee-jerk optimization.
• The mapping converges to a critical point of $ Z $ on $ \Sigma $, and since $ Z $ is log-concave, this point is the unique global maximum.
• Generalizations to $ \Sigma_{{\bf a}} $ and products of simplices preserve the monotonicity property via weighted and component-wise I-divergence terms.