[论文解读] An improved analysis of the ER-SpUD dictionary learning algorithm

Jarosław Błasiok, Jelani Nelson|arXiv (Cornell University)|Feb 18, 2016

Sparse and Compressive Sensing Techniques参考文献 27被引用 6

一句话总结

本文提出了对ER-SpUD字典学习算法的改进理论分析，表明通过一个微小修改，可在 $ p \gtrsim n\log(n/\delta) $ 个样本下实现字典的可靠恢复，解决了 [SWW12] 的猜想，并反驳了先前认为原始算法在 $ p \gtrsim n\log^4 n $ 样本下即足够有效的说法。关键结果在次高斯稀疏性和无噪声条件下建立了紧致的样本复杂度边界。

ABSTRACT

In "dictionary learning" we observe $Y = AX + E$ for some $Y\in\mathbb{R}^{n imes p}$, $A \in\mathbb{R}^{m imes n}$, and $X\in\mathbb{R}^{m imes p}$. The matrix $Y$ is observed, and $A, X, E$ are unknown. Here $E$ is "noise" of small norm, and $X$ is column-wise sparse. The matrix $A$ is referred to as a {\em dictionary}, and its columns as {\em atoms}. Then, given some small number $p$ of samples, i.e.\ columns of $Y$, the goal is to learn the dictionary $A$ up to small error, as well as $X$. The motivation is that in many applications data is expected to sparse when represented by atoms in the "right" dictionary $A$ (e.g.\ images in the Haar wavelet basis), and the goal is to learn $A$ from the data to then use it for other applications. Recently, [SWW12] proposed the dictionary learning algorithm ER-SpUD with provable guarantees when $E = 0$ and $m = n$. They showed if $X$ has independent entries with an expected $s$ non-zeroes per column for $1 \lesssim s \lesssim \sqrt{n}$, and with non-zero entries being subgaussian, then for $p\gtrsim n^2\log^2 n$ with high probability ER-SpUD outputs matrices $A', X'$ which equal $A, X$ up to permuting and scaling columns (resp.\ rows) of $A$ (resp.\ $X$). They conjectured $p\gtrsim n\log n$ suffices, which they showed was information theoretically necessary for {\em any} algorithm to succeed when $s \simeq 1$. Significant progress was later obtained in [LV15]. We show that for a slight variant of ER-SpUD, $p\gtrsim n\log(n/δ)$ samples suffice for successful recovery with probability $1-δ$. We also show that for the unmodified ER-SpUD, $p\gtrsim n^{1.99}$ samples are required even to learn $A, X$ with polynomially small success probability. This resolves the main conjecture of [SWW12], and contradicts the main result of [LV15], which claimed that $p\gtrsim n\log^4 n$ guarantees success whp.

研究动机与目标

• 解决 [SWW12] 的猜想，即在次高斯稀疏性条件下，$ p \gtrsim n\log n $ 个样本足以通过ER-SpUD实现字典恢复。
• 分析ER-SpUD改进版本的样本复杂度，并在高概率恢复条件下建立紧致边界。
• 证明原始ER-SpUD算法即使在成功概率为多项式小量时，也需 $ p \gtrsim n^{1.99} $ 个样本才能成功，从而反驳 [LV15] 中的声称。
• 在无噪声的稀疏、次高斯系数模型下，提供字典学习更精细的理论理解。

提出的方法

• 引入ER-SpUD的微小变体，以改善样本复杂度的保证。
• 利用泛化链和高斯过程尾部界，控制 $ \ell_1 $-球上随机过程的上确界。
• 通过度量熵和主导测度理论，对Rademacher混沌过程的期望上确界进行有界。
• 应用浓度不等式，推导经验范数与其期望之间偏差的高概率界。
• 推导在 $ \ell_2 $ 和 $ \ell_\infty $ 范数下 $ \ell_1 $-球的 $ \gamma_2 $ 和 $ \gamma_1 $ 泛函的界。
• 通过引理14和引理24结合矩不等式与尾部界，将浓度行为从一个过程转移到其对称化版本。

实验结果

研究问题

• RQ1 在次高斯稀疏系数下，$ p \gtrsim n\log n $ 个样本是否足以实现字典学习的高概率恢复？
• RQ2 原始ER-SpUD算法是否如 [LV15] 所声称的那样，可在 $ p \gtrsim n\log^4 n $ 个样本下实现高概率恢复？
• RQ3 原始ER-SpUD算法即使在成功概率为多项式小量时，其最小样本复杂度是多少？
• RQ4 所提出的ER-SpUD变体相比原始算法在样本复杂度上如何改进？
• RQ5 度量熵和泛化链在紧致分析字典学习算法中起到何种作用？

主要发现

• 改进后的ER-SpUD版本可在 $ p \gtrsim n\log(n/\delta) $ 个样本下实现高概率字典恢复，解决了 [SWW12] 的猜想。
• 原始ER-SpUD算法即使在成功概率为 $ 1/\mathop{\mathrm{poly}}(n) $ 的情况下，也需 $ p \gtrsim n^{1.99} $ 个样本才能成功，与 [LV15] 的声称相矛盾。
• 分析表明，$ \gamma_2(B_1, \|\cdot\|_2) \lesssim \sqrt{\log n} $ 且 $ \gamma_1(B_1, \|\cdot\|_\infty) \lesssim \log n $，这对控制过程上确界至关重要。
• 高概率下，对称化过程 $ \sup_{v \in B_1} |\tilde{X}_v| $ 的上界为 $ \lesssim \sqrt{m\theta \log(n/\delta)} + \log(n/\delta) $。
• 在给定的稀疏性和次高斯假设下，所需样本量 $ p \gtrsim n\log(n/\delta) $ 对于改进算法而言既是必要也是充分的。
• 结果表明，当稀疏度 $ \theta \simeq 1/n $ 时，$ p \gtrsim n\log n $ 在信息论上是恢复所必需且充分的，从而证实了该猜想。