[Paper Review] Online Multi-task Learning with Hard Constraints

Gábor Lugosi, Omiros Papaspiliopoulos+1 more|ArXiv.org|Feb 20, 2009

Advanced Bandit Algorithms Research19 references21 citations

TL;DR

This paper introduces an online multi-task learning framework with hard constraints on simultaneous actions across M tasks, reducing the problem to an online shortest path problem for efficient computation. It achieves low regret with computationally tractable algorithms, extending to tracking, bandit feedback, and infinite-task settings with provable bounds and efficient approximation via discretization.

ABSTRACT

We discuss multi-task online learning when a decision maker has to deal simultaneously with M tasks. The tasks are related, which is modeled by imposing that the M-tuple of actions taken by the decision maker needs to satisfy certain constraints. We give natural examples of such restrictions and then discuss a general class of tractable constraints, for which we introduce computationally efficient ways of selecting actions, essentially by reducing to an on-line shortest path problem. We briefly discuss "tracking" and "bandit" versions of the problem and extend the model in various ways, including non-additive global losses and uncountably infinite sets of tasks.

Motivation & Objective

• To develop a computationally efficient online learning strategy for multiple related tasks under hard constraints on joint actions.
• To minimize regret across all tasks while respecting global constraints that restrict feasible action combinations.
• To extend the framework to tracking (limited action changes) and bandit feedback (partial loss observation).
• To handle uncountably infinite tasks by modeling actions as functions and using approximation techniques.
• To provide theoretical regret bounds with practical implementation complexity.

Proposed method

• Reduces the constrained multi-task problem to an online shortest path problem in a constructed graph, enabling efficient computation using known algorithms.
• Uses exponentially weighted average forecasters for robust performance, especially in bandit feedback settings.
• Applies a discretization scheme by partitioning the task space into intervals, aggregating tasks into super-tasks to enable finite computation.
• Employs importance sampling and particle filtering for approximate sampling in continuous task settings, with stability guarantees.
• Introduces a modified implementation that improves efficiency over standard online shortest path solvers.
• Extends results to non-additive global losses and continuous-time Markov chain approximations for infinite tasks.

Experimental results

Research questions

• RQ1 How can online multi-task learning be efficiently performed when joint actions must satisfy hard constraints?
• RQ2 Can the computational complexity of constrained multi-task learning be reduced to that of an online shortest path problem?
• RQ3 How can the framework be adapted to tracking (limited action changes) and bandit feedback (partial loss observation)?
• RQ4 What approximation techniques enable efficient implementation in infinite-task settings?
• RQ5 What theoretical regret bounds can be achieved under these constraints and feedback models?

Key findings

• The regret is bounded by $ \sqrt{\frac{nm\ln(N\lceil 1/\varepsilon\rceil)}{2}} + \frac{mn\varepsilon}{2} + \sqrt{\frac{n}{2}\ln\frac{1}{\delta}} $ with high probability, for any $ \varepsilon > 0 $.
• With $ \varepsilon \sim 1/\sqrt{n} $, the regret bound matches that of the finite case, achieving $ O(\sqrt{n}) $ growth.
• The computational complexity is $ O\bigl{(}(Nm)^{2}/\varepsilon\bigr{)} $, which becomes $ O\bigl{(}\sqrt{n}(Nm)^{2}\bigr{)} $ for optimal $ \varepsilon $.
• The method extends naturally to bandit feedback, where only the sum of losses over chosen actions is observed.
• Approximation via discretization achieves $ O(\varepsilon) $ error in loss summation, with total variation error $ O((m+1)/K) $ in particle filtering.
• The framework supports infinite-task settings by modeling actions as functions and using Feynman-Kac formulae with sequential Monte Carlo.