[论文解读] Non-Convex Projected Gradient Descent for Generalized Low-Rank Tensor Regression

Han Chen, Garvesh Raskutti+1 more|arXiv (Cornell University)|Nov 30, 2016

Sparse and Compressive Sensing Techniques被引用 41

一句话总结

本文提出了一种用于广义低秩张量回归的非凸投影梯度下降（PGD）框架，通过局部高斯宽度建立理论保证。结果表明，在低秩张量结构下，PGD在统计误差率和收敛速度方面优于凸松弛方法，展现出可证明的线性收敛性，并在三种张量秩模型下实现了改进的样本复杂度：切片秩之和、稀疏-低秩切片以及Tucker秩。

ABSTRACT

In this paper, we consider the problem of learning high-dimensional tensor regression problems with low-rank structure. One of the core challenges associated with learning high-dimensional models is computation since the underlying optimization problems are often non-convex. While convex relaxations could lead to polynomial-time algorithms they are often slow in practice. On the other hand, limited theoretical guarantees exist for non-convex methods. In this paper we provide a general framework that provides theoretical guarantees for learning high-dimensional tensor regression models under different low-rank structural assumptions using the projected gradient descent algorithm applied to a potentially non-convex constraint set $\\Theta$ in terms of its \\emph{localized Gaussian width}. We juxtapose our theoretical results for non-convex projected gradient descent algorithms with previous results on regularized convex approaches. The two main differences between the convex and non-convex approach are: (i) from a computational perspective whether the non-convex projection operator is computable and whether the projection has desirable contraction properties and (ii) from a statistical upper bound perspective, the non-convex approach has a superior rate for a number of examples. We provide three concrete examples of low-dimensional structure which address these issues and explain the pros and cons for the non-convex and convex approaches. We supplement our theoretical results with simulations which show that, under several common settings of generalized low rank tensor regression, the projected gradient descent approach is superior both in terms of statistical error and run-time provided the step-sizes of the projected descent algorithm are suitably chosen.

研究动机与目标

• 开发一种用于高维张量回归的非凸优化框架，以克服凸松弛方法的计算瓶颈。
• 在一般低秩张量约束下，建立非凸PGD的理论收敛性与统计误差界。
• 比较非凸PGD与凸正则化方法在张量回归中的统计性能与计算性能。
• 形式化非凸投影产生压缩性质与快速收敛的条件。
• 通过三个具体的低秩张量模型，展示改进的误差率与运行时效率。

提出的方法

• 该方法在表示低秩张量结构的非凸约束集 Θ 上应用投影梯度下降，投影操作满足压缩性质。
• 提出一种基于对称锥的超可加族与受控压缩的近似投影的一般框架。
• 理论风险界以 Θ ∩ B_F(1) 的局部高斯宽度表示，其中 B_F(1) 为半径为1的Frobenius范数球。
• 该框架适用于三种张量秩模型：切片秩之和、稀疏性与低秩切片、以及Tucker秩。
• 通过递归矩阵化与奇异值阈值算子，证明了收敛性，其投影误差被有效界控。
• 统计误差被证明与 n^{-1/2} w_G[Θ ∩ B_F(1)] 成比例，且利用核范数正则化器推导出显式上界。

实验结果

研究问题

• RQ1 非凸投影梯度下降能否在低秩张量回归中实现优于凸正则化方法的统计误差率？
• RQ2 在何种条件下，非凸PGD算法能实现可证明误差界下的线性收敛？
• RQ3 在高维张量设置下，非凸PGD的计算效率与凸松弛方法相比如何？
• RQ4 局部高斯宽度在刻画非凸PGD用于张量模型的统计误差中起什么作用？
• RQ5 不同的低秩张量结构（如Tucker秩、切片秩）如何影响PGD的收敛性与误差率？

主要发现

• 非凸PGD方法实现了 n^{-1/2} w_G[Θ ∩ B_F(1)] 量级的统计误差率，这是张量回归中非凸PGD的首个此类通用上界。
• 对于切片秩之和模型，误差率被界为 O(n^{-1/2} √{(s′+s)(r′+r)} √{6(d₁+d₂+log d₃)} )。
• 对于稀疏-低秩切片模型，误差率被界为 O(n^{-1/2} √{(r′+r)(s′+s)} √{6(d₁+d₂+log d₃)} )。
• 对于Tucker秩模型，误差率被界为 O(n^{-1/2} √{r′+r} √{6 min{d₁+d₂d₃, d₂+d₁d₃, d₃+d₁d₂}} )。
• 与凸正则化方案相比，非凸PGD方法在统计误差率上表现更优，最多仅差一个常数因子，与Raskutti和Yuan（2015）的定理1对比可得。
• 模拟结果证实，当步长适当调优时，非凸PGD在统计误差与运行时间方面均优于凸方法。