[Paper Review] A Generalization Bound of Deep Neural Networks for Dependent Data

Quan Do, Binh T. Nguyen+1 more|arXiv (Cornell University)|Oct 9, 2023

Stochastic Gradient Optimization TechniquesComputer Science3 citations

TL;DR

This paper establishes a generalization bound for feed-forward deep neural networks under non-iid, non-stationary $φ$-mixing data, relaxing the standard i.i.d. assumption. By leveraging Rademacher complexity and accounting for weak dependence and time-varying marginal distributions, it derives a generalization error bound that extends prior results to dependent data settings such as epidemiology and finance.

ABSTRACT

Existing generalization bounds for deep neural networks require data to be independent and identically distributed (iid). This assumption may not hold in real-life applications such as evolutionary biology, infectious disease epidemiology, and stock price prediction. This work establishes a generalization bound of feed-forward neural networks for non-stationary $ϕ$-mixing data.

Motivation & Objective

• Address the lack of generalization theory for deep neural networks when data are dependent and non-stationary, a common scenario in real-world applications.
• Relax the i.i.d. assumption in existing generalization bounds to accommodate $φ$-mixing sequences where dependence decays over time.
• Allow for non-identically distributed data where marginal distributions converge to a target distribution at a sub-Gaussian rate.
• Derive a uniform generalization bound for feed-forward neural networks under these relaxed stochastic assumptions.
• Enable theoretical guarantees in domains like evolutionary biology, disease modeling, and financial time series where data dependence is inherent.

Proposed method

• Formalize the data generation process as a non-stationary $φ$-mixing stochastic process, where dependence between observations diminishes with increasing time separation.
• Use Rademacher complexity to bound the difference between expected and empirical loss over a hypothesis space of feed-forward neural networks.
• Introduce a novel bound that incorporates the mixing coefficient decay rate $φ(n) = Ø(1/n)$ and the total variation distance $μ_n = Ø(1/√{n})$ between empirical and target distributions.
• Apply a uniform bound over network parameters using a union bound over weight matrices with bounded spectral and $\ell_{2,1}$-norms.
• Derive a Rademacher complexity bound for ReLU-like networks with $p_i$-Lipschitz activations and bounded weight matrices.
• Combine the generalization bound with a margin-based error decomposition to relate the classification error to the empirical loss and complexity terms.

Experimental results

Research questions

• RQ1 Can generalization bounds for deep neural networks be extended to non-iid, non-stationary data such as time series with weak dependence?
• RQ2 How does the decay of dependence in $φ$-mixing sequences affect the generalization error of deep neural networks?
• RQ3 What is the impact of time-varying marginal distributions on the convergence of empirical risk to expected risk?
• RQ4 Can Rademacher complexity be adapted to bound generalization error under weak dependence and non-identical distributions?
• RQ5 Does the proposed bound reduce to the standard i.i.d. case when the data become independent and identically distributed?

Key findings

• The generalization bound is derived under a non-iid, non-stationary $φ$-mixing assumption, allowing for time-dependent data with decaying dependence.
• The bound includes a term $\frac{1}{n}\sum_{i=1}^{n}\mu_i = \mathcal{O}(1/\sqrt{n})$, capturing the convergence of empirical marginal distributions to the target distribution.
• The term $||\Delta_n||_{\infty} = \mathcal{O}(\log n)$ accounts for the dependence structure through the mixing coefficient, reflecting the impact of weak dependence on generalization.
• The Rademacher complexity term scales as $\mathcal{O}\left(\frac{\ln n}{n}\right)$, with explicit dependence on network depth, weight norms, and activation Lipschitz constants.
• The final generalization error bound includes a $\mathcal{O}(1/\sqrt{n})$ term from distributional shift and a $\mathcal{O}(\log n / n)$ term from mixing dependence.
• The bound recovers the i.i.d. case of Bartlett et al. (2017) as a special case when $\varphi(n) \to 0$ and $\mu_n \to 0$ rapidly enough.