[Paper Review] Online Learning: A Comprehensive Survey

Steven C. H. Hoi, Doyen Sahoo+2 more|arXiv (Cornell University)|Feb 8, 2018

Data Stream Mining Techniques273 references144 citations

TL;DR

This survey systematically reviews online learning literature, focusing on online supervised learning and partial-feedback settings, with theoretical foundations and taxonomy.

ABSTRACT

Online learning represents an important family of machine learning algorithms, in which a learner attempts to resolve an online prediction (or any type of decision-making) task by learning a model/hypothesis from a sequence of data instances one at a time. The goal of online learning is to ensure that the online learner would make a sequence of accurate predictions (or correct decisions) given the knowledge of correct answers to previous prediction or learning tasks and possibly additional information. This is in contrast to many traditional batch learning or offline machine learning algorithms that are often designed to train a model in batch from a given collection of training data instances. This survey aims to provide a comprehensive survey of the online machine learning literatures through a systematic review of basic ideas and key principles and a proper categorization of different algorithms and techniques. Generally speaking, according to the learning type and the forms of feedback information, the existing online learning works can be classified into three major categories: (i) supervised online learning where full feedback information is always available, (ii) online learning with limited feedback, and (iii) unsupervised online learning where there is no feedback available. Due to space limitation, the survey will be mainly focused on the first category, but also briefly cover some basics of the other two categories. Finally, we also discuss some open issues and attempt to shed light on potential future research directions in this field.

Motivation & Objective

• Summarize core ideas, principles, and taxonomy of online learning.
• Review online learning foundations from learning theory, optimization, and game theory.
• Highlight online supervised learning and partial-feedback settings while briefly touching unsupervised learning.
• Discuss open issues and future research directions in online learning.

Proposed method

• Classify online learning techniques by feedback type: online supervised, limited feedback, and unsupervised.
• Present standard problem formulations for online binary classification and regret minimization.
• Explain empirical risk minimization, excess risk decomposition, and online convex optimization frameworks.
• Describe major algorithmic families: first-order, second-order, and regularization-based methods (OGD, ONS, FTRL, OMD, EG, AdaGrad).
• Outline connections to game theory and repeated zero-sum games in online learning contexts.

Experimental results

Research questions

• RQ1 What are the main categories and feedback models in online learning?
• RQ2 What are the foundational theories (learning theory, optimization, game theory) underpinning online learning?
• RQ3 What are the core online convex optimization methods and their regret guarantees?
• RQ4 How do online learning algorithms relate to classical batch learning and data streams?
• RQ5 What open issues and future directions exist in online learning research?

Key findings

• Provides a taxonomy of online learning methods based on feedback: online supervised, online learning with limited feedback, and online unsupervised learning.
• Defines and discusses regret, empirical risk, and the bias-variance trade-off in online learning contexts.
• Reviews first-order, second-order, and regularization-based online optimization algorithms and their theoretical guarantees.
• Connects online learning to game theory, including Nash equilibria and minimax concepts in repeated zero-sum games.
• Points to open issues and potential future research directions in online learning and online convex optimization.