[Paper Review] Learning from dependent observations

Ingo Steinwart, Don Hush+1 more|ArXiv.org|Jul 2, 2007

Machine Learning and Algorithms35 references4 citations

TL;DR

This paper establishes the consistency of support vector machines (SVMs) under general dependence assumptions by showing that SVMs remain consistent as long as the data-generating process satisfies a law of large numbers. The key contribution is proving consistency for SVMs with a-priori chosen regularization parameters under $α$-mixing (non-stationary) processes, even with unbounded noise and without requiring compactness of the input space.

ABSTRACT

In most papers establishing consistency for learning algorithms it is assumed that the observations used for training are realizations of an i.i.d. process. In this paper we go far beyond this classical framework by showing that support vector machines (SVMs) essentially only require that the data-generating process satisfies a certain law of large numbers. We then consider the learnability of SVMs for $\a$-mixing (not necessarily stationary) processes for both classification and regression, where for the latter we explicitly allow unbounded noise.

Motivation & Objective

• To extend the theoretical justification of SVMs beyond the i.i.d. assumption, which is often violated in real-world temporal and heterogeneous data.
• To establish consistency of SVMs for non-stationary, $α$-mixing processes, where dependence decays polynomially.
• To show that SVMs remain consistent even when the input space is non-compact and noise is unbounded.
• To demonstrate that a regularization sequence can be chosen a-priori for $α$-mixing processes, avoiding adaptive tuning based on unknown process properties.
• To define a meaningful risk functional and limit distribution for non-i.i.d. processes using laws of large numbers.

Proposed method

• The paper introduces a generalized law of large numbers (LLN) framework for non-stationary stochastic processes, enabling the definition of a limit distribution and a corresponding risk functional.
• It defines consistency of learning algorithms relative to this limit distribution for processes satisfying the LLN.
• The core analysis uses kernel methods and representer theorems to express SVM solutions in a reproducing kernel Hilbert space (RKHS), with regularization parameters $\lambda_n$ controlling the trade-off between empirical risk and smoothness.
• The proof relies on bounding the difference between the true risk and the empirical risk using moment inequalities and mixing coefficients, particularly $\alpha$-mixing rates.
• It applies moment bounds and exponential tail inequalities to control the deviation of the empirical mean from the true mean under dependence.
• The analysis extends to unbounded noise by using $L_p$-type loss functions and controlling the growth of the solution norm via $\lambda_n$-dependent bounds.

Experimental results

Research questions

• RQ1 Can SVMs be consistently trained when the training data are not i.i.d., but instead follow a weakly dependent, non-stationary process?
• RQ2 Is it possible to choose the regularization parameter $\lambda_n$ a-priori (without knowledge of the process) for consistent SVM learning under dependence?
• RQ3 Does the compactness assumption on the input space $X$ remain necessary for SVM consistency under dependent data?
• RQ4 Can SVMs achieve consistency when the noise in regression is unbounded, under dependence?
• RQ5 What conditions on the dependence structure (e.g., $\alpha$-mixing) ensure that the empirical risk converges to the true risk in expectation?

Key findings

• SVMs are consistent for any data-generating process satisfying a generalized law of large numbers, even if the process is non-stationary and dependent.
• For $\alpha$-mixing processes with polynomially decaying coefficients, a regularization sequence $\lambda_n$ can be chosen a-priori to ensure consistency.
• The compactness assumption on the input space $X$ is not required for SVM consistency when using Gaussian kernels.
• The paper establishes consistency for SVMs with unbounded noise in regression, extending prior results that required bounded noise.
• The convergence rate of the empirical risk to the true risk is controlled by the mixing rate and the choice of $\lambda_n$, with the deviation probability decaying as $O(\lambda_n^{-2p} n^{-\beta})$ for $\beta > 0$.
• The analysis provides a theoretical foundation for the empirical success of SVMs in non-i.i.d. settings such as time series, speech recognition, and system diagnosis.