[论文解读] Learning From An Optimization Viewpoint

Karthik Sridharan|arXiv (Cornell University)|Apr 18, 2012

Machine Learning and Algorithms参考文献 84被引用 19

一句话总结

该论文将机器学习重新定义为优化问题，表明在一般学习设置中，传统的统一收敛方法（如经验风险最小化，ERM）可能失效，而随机逼近（SA）方法则能成功。论文引入了序列覆盖数与序列打包数以刻画可学习性与可计算性，证明序列复杂度度量（如序列扇形破碎维数）相较于经典的VC型度量，能为非独立同分布或结构化数据提供更紧的界。

ABSTRACT

In this dissertation we study statistical and online learning problems from an optimization viewpoint.The dissertation is divided into two parts : I. We first consider the question of learnability for statistical learning problems in the general learning setting. The question of learnability is well studied and fully characterized for binary classification and for real valued supervised learning problems using the theory of uniform convergence. However we show that for the general learning setting uniform convergence theory fails to characterize learnability. To fill this void we use stability of learning algorithms to fully characterize statistical learnability in the general setting. Next we consider the problem of online learning. Unlike the statistical learning framework there is a dearth of generic tools that can be used to establish learnability and rates for online learning problems in general. We provide online analogs to classical tools from statistical learning theory like Rademacher complexity, covering numbers, etc. We further use these tools to fully characterize learnability for online supervised learning problems. II. In the second part, for general classes of convex learning problems, we provide appropriate mirror descent (MD) updates for online and statistical learning of these problems. Further, we show that the the MD is near optimal for online convex learning and for most cases, is also near optimal for statistical convex learning. We next consider the problem of convex optimization and show that oracle complexity can be lower bounded by the so called fat-shattering dimension of the associated linear class. Thus we establish a strong connection between offline convex optimization problems and statistical learning problems. We also show that for a large class of high dimensional optimization problems, MD is in fact near optimal even for convex optimization.

研究动机与目标

• 将统计学习与在线学习重新定义为优化问题，以深化对学习、优化与泛化之间联系的理解。
• 研究经典统一收敛理论（如VC维、Rademacher复杂度）在刻画一般学习问题可学习性方面的局限性。
• 基于序列覆盖数与序列打包数，建立一个新理论框架，用于分析统计学习与在线学习中的可学习性与可计算性。
• 证明即使在凸设置下，随机逼近（SA）也能在经验风险最小化（ERM）失效时提供学习保证。
• 利用序列复杂度度量（如序列扇形破碎维数）刻画凸学习问题的Oracle复杂度与收敛速率。

提出的方法

• 将学习问题形式化为随机优化问题，区分经验风险最小化（样本平均近似）与随机逼近（SA）方法。
• 引入序列覆盖数 $ N^\text{seq}_p(\alpha, \mathcal{F}, z) $，作为在深度为 $ n $ 的树上函数类复杂度的度量，捕捉路径依赖行为。
• 定义两种打包方式：弱打包 $ D_p(\alpha, \mathcal{F}, z) $ 与强打包 $ M_p(\alpha, \mathcal{F}, z) $，后者要求在共同路径上保持分离。
• 建立不等式 $ M_p(2\alpha, \mathcal{F}, z) \leq N^\text{seq}_p(\alpha, \mathcal{F}, z) \leq D_p(\alpha, \mathcal{F}, z) $，将序列设置下的覆盖与打包联系起来。
• 证明组合界：$ N^\text{seq}_\infty(1/2, \mathcal{F}, n) \leq \sum_{i=0}^d \binom{n}{i} k^i \leq (ekn)^d $，其中 $ d = \text{fat}^\text{seq}_2(\mathcal{F}) $，将Sauer–Shelah引理由树结构推广。
• 通过离散化与序列复杂度，界定了凸学习问题中的Oracle复杂度与收敛速率。

实验结果

研究问题

• RQ1 为何统一收敛理论无法刻画一般统计学习问题中的可学习性？
• RQ2 即使在凸设置下，随机逼近（SA）是否能在经验风险最小化（ERM）失效时仍提供学习保证？
• RQ3 序列覆盖数与序列打包数与经典VC或基于Rademacher的度量在刻画函数类复杂度方面有何不同？
• RQ4 序列扇形破碎维数在非独立同分布或结构化设置中，对可学习性与收敛速率起何作用？
• RQ5 凸学习问题的Oracle复杂度与序列复杂度度量（如 $ N^\text{seq}_p(\alpha, \mathcal{F}, z) $）有何关联？

主要发现

• 构造了一个反例，表明在凸学习问题中，SA可实现成功学习，而ERM无法提供任何有意义的泛化保证。
• 序列覆盖数 $ N^\text{seq}_\infty(1/2, \mathcal{F}, n) $ 的上界为 $ (ekn)^d $，其中 $ d = \text{fat}^\text{seq}_2(\mathcal{F}) $，将Sauer–Shelah引理由树结构推广至序列函数类。
• 弱打包数与强打包数之间的差距可高达 $ 2^n $，凸显了在序列设置中路径特定分离的重要性。
• 不等式 $ M_p(2\alpha, \mathcal{F}, z) \leq N^\text{seq}_p(\alpha, \mathcal{F}, z) \leq D_p(\alpha, \mathcal{F}, z) $ 建立了序列覆盖与打包之间的紧密联系，从而支持新的泛化界。
• 该框架表明，序列复杂度度量（如序列扇形破碎维数）相较于经典度量，更适合分析在线学习与非独立同分布学习问题。
• 结果表明，基于优化的学习（通过SA）可在经典ERM方法失效的场景中成功，即使问题为凸。