[Paper Review] Prediction with expert evaluators' advice

Alexey Chernov, Vladimir Vovk|ArXiv.org|Feb 24, 2009

Advanced Bandit Algorithms Research9 references6 citations

TL;DR

This paper introduces a novel prediction framework where each expert evaluates the learner using their own time-varying, potentially different loss functions. It shows that the defensive forecasting algorithm achieves the same optimal performance guarantee as the Aggregating Algorithm in standard settings when all loss functions are proper and mixable, and extends this to specialist (sleeping) experts via a modified algorithm with normalized weights and selective weight updates.

ABSTRACT

We introduce a new protocol for prediction with expert advice in which each expert evaluates the learner's and his own performance using a loss function that may change over time and may be different from the loss functions used by the other experts. The learner's goal is to perform better or not much worse than each expert, as evaluated by that expert, for all experts simultaneously. If the loss functions used by the experts are all proper scoring rules and all mixable, we show that the defensive forecasting algorithm enjoys the same performance guarantee as that attainable by the Aggregating Algorithm in the standard setting and known to be optimal. This result is also applied to the case of "specialist" (or "sleeping") experts. In this case, the defensive forecasting algorithm reduces to a simple modification of the Aggregating Algorithm.

Motivation & Objective

• To address the limitation in standard prediction with expert advice where a single, fixed loss function is used for all comparisons.
• To enable each expert to evaluate the learner using their own personalized, time-varying loss function.
• To maintain the same optimal performance guarantee as the Aggregating Algorithm under proper and mixable loss functions.
• To extend the framework to specialist (sleeping) experts, where not all experts are active at each step.
• To provide a modified defensive forecasting algorithm that adapts to expert availability and maintains theoretical performance bounds.

Proposed method

• Introduces a new protocol where each expert uses their own loss function to evaluate the learner’s performance, which may vary over time and differ across experts.
• Applies the defensive forecasting algorithm, which selects predictions based on weighted combinations of experts’ predictions using normalized weights.
• Uses a substitution function Σ that ensures the learner’s prediction satisfies a convexity condition in the exponential of the loss, derived from mixability.
• For specialist experts, the algorithm updates weights only for active experts based on the difference between learner and expert losses.
• Normalizes weights at each step using only the currently awake experts to maintain consistency in the prediction rule.
• Employs a performance guarantee based on the mixability parameter η, ensuring the learner’s cumulative loss stays within 1/η ln N of each expert’s loss.

Experimental results

Research questions

• RQ1 Can a learner achieve performance comparable to each expert when each expert evaluates the learner using their own distinct, time-varying loss function?
• RQ2 Does the defensive forecasting algorithm maintain the same optimal performance guarantee as the Aggregating Algorithm when loss functions are proper and mixable but not necessarily identical across experts?
• RQ3 How can the framework be adapted to handle specialist (sleeping) experts, where only a subset of experts are active at each time step?
• RQ4 What modifications to the defensive forecasting algorithm are required to maintain theoretical performance bounds in the presence of expert inactivity?
• RQ5 Is the performance guarantee of the defensive forecasting algorithm preserved when loss functions are proper scoring rules and mixable, even when they differ across experts?

Key findings

• The defensive forecasting algorithm achieves the same performance guarantee as the Aggregating Algorithm in the standard setting, with the learner’s cumulative loss bounded by L_t ≤ L_t^n + (1/η) ln N for all experts n and all t.
• This guarantee holds when all loss functions used by experts are proper scoring rules and η-mixable, ensuring optimal performance under the new protocol.
• For specialist experts, the defensive forecasting algorithm reduces to a modified version that updates weights only for awake experts, preserving the performance bound.
• The algorithm uses normalized weights u_t^n = w_t^n / sum_{n∈A_t} w_t^n to ensure consistency in the prediction rule across time steps.
• The weight update rule w_t^n := w_{t-1}^n exp(η(λ(π_t, ω_t) - λ(γ_t^n, ω_t))) ensures that better-performing experts (relative to the learner) gain weight, while worse ones lose weight.
• The framework generalizes to non-binary outcomes, arbitrary prediction spaces, and countably many experts, provided the underlying defensive forecasting method applies.