[Paper Review] The Second Order Linear Model

Ming Lin, Shuang Qiu+2 more|arXiv (Cornell University)|Mar 2, 2017

Stochastic Gradient Optimization Techniques13 references3 citations

TL;DR

This paper proposes the moment-estimation-sequence (MES) method, a gradient-free, nonconvex alternating algorithm for learning the second-order linear model (SLM) with $O(k^2d)$ sampling complexity. It achieves global, linear convergence by eliminating gradient bias from skewed sub-Gaussian distributions, outperforming conventional gradient descent in both convergence speed and robustness on non-MIP distributions.

ABSTRACT

We study a fundamental class of regression models called the second order linear model (SLM). The SLM extends the linear model to high order functional space and has attracted considerable research interest recently. Yet how to efficiently learn the SLM under full generality using nonconvex solver still remains an open question due to several fundamental limitations of the conventional gradient descent learning framework. In this study, we try to attack this problem from a gradient-free approach which we call the moment-estimation-sequence (MES) method. We show that the conventional gradient descent heuristic is biased by the skewness of the distribution therefore is no longer the best practice of learning the SLM. Based on the MES framework, we design a nonconvex alternating iteration process to train a $d$-dimension rank-$k$ SLM within $O(kd)$ memory and one-pass of the dataset. The proposed method converges globally and linearly, achieves $ε$ recovery error after retrieving $O[k^{2}d\cdot\mathrm{polylog}(kd/ε)]$ samples. Furthermore, our theoretical analysis reveals that not all SLMs can be learned on every sub-gaussian distribution. When the instances are sampled from a so-called $τ$-MIP distribution, the SLM can be learned by $O(p/τ^{2})$ samples where $p$ and $τ$ are positive constants depending on the skewness and kurtosis of the distribution. For non-MIP distribution, an addition diagonal-free oracle is necessary and sufficient to guarantee the learnability of the SLM. Numerical simulations verify the sharpness of our bounds on the sampling complexity and the linear convergence rate of our algorithm.

Motivation & Objective

• To address the open problem of efficient, provable nonconvex learning of the second-order linear model (SLM) under general sub-Gaussian distributions.
• To overcome the fundamental limitation of gradient descent in skewed distributions, where gradients become biased by high-order moments.
• To develop a memory-efficient, one-pass algorithm with $O(kd)$ memory and $O(k^2d)$ sampling complexity for low-rank SLMs.
• To establish theoretical conditions under which SLMs are learnable, distinguishing between $\tau$-MIP and non-MIP distributions.
• To demonstrate that a diagonal-free oracle is necessary and sufficient for learnability on non-MIP distributions.

Proposed method

• Proposes the moment-estimation-sequence (MES) framework, a gradient-free approach that constructs a sequence of moment estimators to avoid gradient bias.
• Designs a nonconvex alternating iteration process that updates the first- and second-order coefficients in a block-optimization manner.
• Employs a one-pass data stream processing strategy to maintain $O(kd)$ memory usage and achieve online learning capability.
• Leverages sub-Gaussian Hanson-Wright inequalities and concentration bounds to analyze the convergence of moment estimators.
• Introduces the $\tau$-MIP (Moment Invertible Property) condition to characterize learnability on general sub-Gaussian distributions.
• Incorporates a diagonal-free oracle for non-MIP distributions, ensuring identifiability of the second-order coefficient matrix $M^*$.

Experimental results

Research questions

• RQ1 Can a nonconvex, gradient-free algorithm achieve global and linear convergence for learning the second-order linear model under general sub-Gaussian distributions?
• RQ2 Why does conventional gradient descent fail in learning SLMs on skewed sub-Gaussian distributions like Bernoulli or truncated Gaussians?
• RQ3 What is the minimal sampling complexity required to recover a rank-$k$ SLM, and how does it depend on distribution properties like skewness and kurtosis?
• RQ4 Under what distributional conditions is the SLM identifiable, and when is an additional diagonal-free oracle necessary?
• RQ5 Can the proposed method achieve $O(k^2d)$ sampling complexity and linear convergence rate, even when the gradient descent heuristic fails?

Key findings

• The MES method achieves global and linear convergence with $O(k^2d \cdot \mathrm{polylog}(kd/\epsilon))$ samples to reach $\epsilon$ recovery error.
• The method converges significantly faster than gradient descent on skewed distributions such as truncated Gaussians and Bernoulli, especially when skewness increases.
• Gradient descent fails to converge globally on $a=0$ truncated Gaussian (high skewness), while MES maintains global convergence.
• For $\tau$-MIP distributions, the SLM is learnable with $O(p/\tau^2)$ samples, where $p$ depends on skewness and kurtosis.
• On non-MIP distributions, a diagonal-free oracle is both necessary and sufficient to guarantee identifiability and learnability of the SLM.
• Numerical simulations confirm the sharpness of the theoretical sampling complexity bounds and validate the linear convergence rate of MES across diverse distributions.