[Paper Review] Online variants of the cross-entropy method

István Szita, András Lörincz|ArXiv.org|Jan 14, 2008

Theoretical and Computational Physics2 references15 citations

TL;DR

This paper introduces two online variants of the cross-entropy method (CEM) for combinatorial optimization, where parameter updates are performed incrementally after each sample using a sliding window or memoryless approach. The authors prove almost sure convergence to the global optimum under mild conditions, extending the theoretical guarantees of batch CEM to online settings with time-invariant distributions.

ABSTRACT

The cross-entropy method is a simple but efficient method for global optimization. In this paper we provide two online variants of the basic CEM, together with a proof of convergence.

Motivation & Objective

• To develop online variants of the cross-entropy method that update parameters incrementally after each sample, rather than in batch after a full population is drawn.
• To establish theoretical convergence guarantees for these online CEM algorithms under the same conditions as the batch version.
• To reduce memory and computational overhead by replacing batch sorting and windowed history with incremental updates or memoryless sampling.
• To demonstrate that online CEM maintains convergence to the global optimum despite the lack of full population knowledge at each step.

Proposed method

• The first online variant uses a sliding window of size $N$ to maintain the last $N$ samples, dynamically computing the elite threshold $\gamma_{t+1}$ as the $\lceil \rho N \rceil$-th highest fitness value in the window.
• After each sample, the algorithm checks if it is among the top $\rho$-percentile of the current window; if so, it performs an online update using a step size $\alpha_1 = \alpha / \lceil \rho N \rceil$.
• The parameter update rule is $\mathbf{p}_{t+1} = (1 - \alpha_1) \mathbf{p}_t + \alpha_1 \mathbf{x}^{(t)}$, applied only when the sample is elite, ensuring incremental adaptation.
• The memoryless variant eliminates the need to store past samples by using a fixed threshold update rule that decreases the elite threshold by a constant $\rho \Delta$ at each step, ensuring eventual detection of elite samples.
• Theoretical analysis uses a recursive probability bound to show that the probability of not having sampled the optimal solution by time $T$ decays exponentially with $T$, under appropriate step size conditions.
• Convergence is proven by showing that the number of parameter updates is infinite almost surely and that the parameter vector eventually stops changing sign, converging to 0 or 1 with probability 1.

Experimental results

Research questions

• RQ1 Can the batch cross-entropy method be adapted into an online, incremental learning framework without sacrificing convergence guarantees?
• RQ2 How can the elite threshold be computed dynamically in an online setting with limited memory and constant per-sample time complexity?
• RQ3 Does the online CEM variant converge almost surely to the global optimum, even when updates are based on partial, sliding windows of past samples?
• RQ4 What modifications are needed to eliminate memory dependence while preserving convergence in online CEM?
• RQ5 How does the use of a fixed decrement in the elite threshold in the memoryless variant affect convergence compared to the sliding window approach?

Key findings

• The online CEM with a sliding window achieves almost sure convergence to the global optimum, as the probability of not having sampled the optimal solution decays exponentially with time.
• The memoryless online CEM variant also converges almost surely, as the elite threshold eventually drops below the minimum fitness value of any non-optimal solution, forcing detection of elite samples.
• The number of parameter updates is infinite almost surely, ensuring that the algorithm continues to refine the distribution parameters indefinitely.
• The parameter vector $\mathbf{p}_t$ converges to 0 or 1 for each component with probability 1, implying convergence to a deterministic solution.
• The convergence rate depends on the step size $\alpha_1$, with smaller $\alpha_1$ leading to faster decay in the probability of not having found the optimum.
• The theoretical framework extends the convergence proof of batch CEM to online settings, establishing that incremental updates preserve the method's global optimization properties.