[Paper Review] Online Continuous DR-Submodular Maximization with Long-Term Budget Constraints

Omid Sadeghi, Maryam Fazel|arXiv (Cornell University)|Jun 30, 2019

Advanced Bandit Algorithms Research23 references4 citations

TL;DR

This paper proposes the Online Saddle Point Hybrid Gradient (OSPHG) algorithm for online continuous DR-submodular maximization under long-term budget constraints, where objective functions are monotone DR-submodular and budget functions are linear. It establishes sub-linear regret and budget violation bounds when the benchmark window size $W = o(T)$, overcoming the impossibility of sub-linear regret under adversarial budget functions with $W = T$. The key contribution is a $(1 - \frac{1}{e})$-approximation regret bound with sub-linear total budget violation for $W = o(T)$.

ABSTRACT

In this paper, we study a class of online optimization problems with long-term budget constraints where the objective functions are not necessarily concave (nor convex) but they instead satisfy the Diminishing Returns (DR) property. Specifically, a sequence of monotone DR-submodular objective functions $\{f_t(x)\}_{t=1}^T$ and monotone linear budget functions $\{\langle p_t,x angle \}_{t=1}^T$ arrive over time and assuming a total targeted budget $B_T$, the goal is to choose points $x_t$ at each time $t\in\{1,\dots,T\}$, without knowing $f_t$ and $p_t$ on that step, to achieve sub-linear regret bound while the total budget violation $\sum_{t=1}^T \langle p_t,x_t angle -B_T$ is sub-linear as well. Prior work has shown that achieving sub-linear regret is impossible if the budget functions are chosen adversarially. Therefore, we modify the notion of regret by comparing the agent against a $(1-\frac{1}{e})$-approximation to the best fixed decision in hindsight which satisfies the budget constraint proportionally over any window of length $W$. We propose the Online Saddle Point Hybrid Gradient (OSPHG) algorithm to solve this class of online problems. For $W=T$, we recover the aforementioned impossibility result. However, when $W=o(T)$, we show that it is possible to obtain sub-linear bounds for both the $(1-\frac{1}{e})$-regret and the total budget violation.

Motivation & Objective

• To address online optimization problems with monotone DR-submodular objectives and linear budget constraints under adversarial conditions.
• To overcome the impossibility of sub-linear regret when budget functions are adversarial over the full horizon ($W = T$).
• To design an algorithm that achieves sub-linear regret and sub-linear budget violation by comparing against a $(1 - \frac{1}{e})$-approximate fixed decision over sliding windows of size $W$.
• To formalize a new regret notion that allows for feasible performance guarantees under time-varying, adversarial budget functions.

Proposed method

• Proposes the Online Saddle Point Hybrid Gradient (OSPHG) algorithm, combining online saddle-point methods with hybrid gradient updates for DR-submodular functions.
• Uses a sliding window of size $W$ to compare the agent's performance against a fixed decision that satisfies the budget constraint proportionally over any window of length $W$.
• Employs a Lagrangian relaxation framework with dual variables $\lambda_t$ to handle budget constraints, where the dual update is based on gradient ascent on the Lagrangian.
• Introduces a hybrid gradient scheme that alternates between primal and dual updates, with convergence analyzed via regret and constraint residual bounds.
• Derives regret and budget violation bounds using a telescoping sum argument and bounding terms via strong convexity and smoothness properties.
• Applies a novel regret decomposition that separates the $(1 - \frac{1}{e})$-regret from the budget violation, enabling sub-linear bounds under $W = o(T)$.

Experimental results

Research questions

• RQ1 Can sub-linear regret be achieved in online continuous DR-submodular maximization when budget functions are adversarial over the full horizon ($W = T$)?
• RQ2 Is it possible to achieve sub-linear total budget violation while maintaining sub-linear regret under adversarial budget functions?
• RQ3 What performance guarantee can be achieved when comparing against a fixed decision that satisfies the budget constraint proportionally over a sliding window of size $W$?
• RQ4 How does the choice of window size $W$ affect the trade-off between regret and budget violation in online DR-submodular optimization?
• RQ5 Can a $(1 - \frac{1}{e})$-approximation regret bound be achieved with sub-linear total budget violation when $W = o(T)$?

Key findings

• Sub-linear regret and sub-linear total budget violation are achievable when the window size $W = o(T)$, overcoming the impossibility result for $W = T$.
• The proposed OSPHG algorithm achieves a $(1 - \frac{1}{e})$-regret bound with sub-linear total budget violation under $W = o(T)$, establishing a new performance guarantee for online DR-submodular optimization.
• The regret and constraint residual bounds are derived using a novel decomposition of the Lagrangian and telescoping sum techniques, with explicit dependence on $W$, $T$, and smoothness parameters.
• The analysis shows that the term $\sum_{t=1}^{T} \lambda_t^2$ can be removed from the final bound when $WT \geq 16R^2$, simplifying the final regret expression.
• The key technical insight is that the window-based benchmark allows cancellation of boundary effects, enabling sub-linear performance even under adversarial budget functions.
• For $W = o(T)$, the algorithm achieves $\mathcal{O}(W)$ regret and $\mathcal{O}(WT)$ budget violation, both sub-linear in $T$, under appropriate parameter tuning.