[Paper Review] Online Learning and Decision-Making under Generalized Linear Model with High-Dimensional Data

Xue Wang, Mike Mingcheng Wei+1 more|arXiv (Cornell University)|Dec 7, 2018

Advanced Bandit Algorithms ResearchDecision Sciences31 references3 citations

TL;DR

This paper proposes the G-MCP-Bandit algorithm, a minimax concave penalized multi-armed bandit method for online decision-making under high-dimensional generalized linear models. It achieves optimal $O(\log T)$ regret in time and $O(\log d)$ in covariate dimension, with a 2-step weighted Lasso procedure enabling accurate estimation under non-iid, sparse data, outperforming benchmarks in synthetic and real-world datasets like warfarin dosing and search advertising.

ABSTRACT

We propose a minimax concave penalized multi-armed bandit algorithm under generalized linear model (G-MCP-Bandit) for a decision-maker facing high-dimensional data in an online learning and decision-making process. We demonstrate that the G-MCP-Bandit algorithm asymptotically achieves the optimal cumulative regret in the sample size dimension T , O(log T), and further attains a tight bound in the covariate dimension d, O(log d). In addition, we develop a linear approximation method, the 2-step weighted Lasso procedure, to identify the MCP estimator for the G-MCP-Bandit algorithm under non-iid samples. Under this procedure, the MCP estimator matches the oracle estimator with high probability and converges to the true parameters with the optimal convergence rate. Finally, through experiments based on synthetic data and two real datasets (warfarin dosing dataset and Tencent search advertising dataset), we show that the G-MCP-Bandit algorithm outperforms other benchmark algorithms, especially when there is a high level of data sparsity or the decision set is large.

Motivation & Objective

• To address the challenge of online learning and decision-making under high-dimensional, sparse data in real-world applications such as personalized medicine and targeted advertising.
• To develop a bandit algorithm that maintains optimal regret scaling in both time $T$ and covariate dimension $d$ under generalized linear models.
• To enable accurate parameter estimation in high-dimensional settings with limited, non-iid samples through a novel 2-step weighted Lasso procedure.
• To ensure the algorithm adapts effectively to data sparsity and large decision sets, outperforming existing benchmarks.

Proposed method

• Proposes the G-MCP-Bandit algorithm, integrating minimax concave penalty (MCP) with multi-armed bandit frameworks for high-dimensional generalized linear models.
• Employs a 2-step weighted Lasso procedure to estimate the MCP estimator under non-i.i.d. sampling, ensuring consistency with high probability.
• Derives theoretical bounds showing the MCP estimator matches the oracle estimator under appropriate regularity conditions.
• Establishes asymptotic optimality of the cumulative regret, achieving $O(\log T)$ in time and $O(\log d)$ in covariate dimension.
• Uses concentration inequalities and matrix norm bounds to control estimation error and gradient conditions in high-dimensional parameter space.
• Applies event-based analysis and high-probability bounds to ensure the oracle estimator is recovered with high probability under sparsity.

Experimental results

Research questions

• RQ1 Can a bandit algorithm achieve optimal $O(\log T)$ regret in high-dimensional online learning under generalized linear models?
• RQ2 How can accurate parameter estimation be achieved under non-i.i.d. and sparse sampling conditions in high-dimensional settings?
• RQ3 Does the proposed 2-step weighted Lasso procedure yield an MCP estimator that matches the oracle estimator with high probability?
• RQ4 How does the G-MCP-Bandit algorithm perform compared to benchmarks in data-sparse or large-decision-set scenarios?
• RQ5 What is the convergence rate of the estimator under the proposed procedure, and does it achieve the optimal rate?

Key findings

• The G-MCP-Bandit algorithm achieves asymptotically optimal cumulative regret of $O(\log T)$ in the time horizon $T$.
• The algorithm attains a tight regret bound of $O\left(\log d\right)$ in the covariate dimension $d$, demonstrating scalability to high-dimensional features.
• The 2-step weighted Lasso procedure ensures the MCP estimator matches the oracle estimator with high probability under non-i.i.d. sampling.
• The estimator converges to the true parameters at the optimal rate, validated through theoretical bounds on estimation error.
• Empirical results show G-MCP-Bandit outperforms benchmark algorithms, especially under data sparsity and large decision sets.
• Theoretical analysis confirms that the gradient condition $\|\nabla_{\mathcal{S}^c}\mathcal{L}(\bm{\beta}^{oracle})\|_\infty \leq \left(1 - \frac{96ns}{|\mathcal{A}|\kappa a}\right)\lambda$ holds with high probability, ensuring model selection consistency.