[论文解读] Learning with tree-based tensor formats

Erwan Grelier, Anthony Nouy+1 more|arXiv (Cornell University)|Nov 11, 2018

Tensor decomposition and applications参考文献 20被引用 13

一句话总结

该论文提出了一种基于树状张量格式的自适应算法，用于学习高维函数，通过多线性优化和随机树结构自适应，实现数值稳定且高精度的近似。该方法优于均匀秩模型，并能以高概率恢复最优树结构，从而在高维统计设置下实现鲁棒且低专家介入的学习。

ABSTRACT

This paper is concerned with the approximation of high-dimensional functions in a statistical learning setting, by empirical risk minimization over model classes of functions in tree-based tensor format. These are particular classes of rank-structured functions that can be seen as deep neural networks with a sparse architecture related to the tree and multilinear activation functions. For learning in a given model class, we exploit the fact that tree-based tensor formats are multilinear models and recast the problem of risk minimization over a nonlinear set into a succession of learning problems with linear models. Suitable changes of representation yield numerically stable learning problems and allow to exploit sparsity. For high-dimensional problems or when only a small data set is available, the selection of a good model class is a critical issue. For a given tree, the selection of the tuple of tree-based ranks that minimize the risk is a combinatorial problem. Here, we propose a rank adaptation strategy which provides in practice a good convergence of the risk as a function of the model class complexity. Finding a good tree is also a combinatorial problem, which can be related to the choice of a particular sparse architecture for deep neural networks. Here, we propose a stochastic algorithm for minimizing the complexity of the representation of a given function over a class of trees with a given arity, allowing changes in the topology of the tree. This tree optimization algorithm is then included in a learning scheme that successively adapts the tree and the corresponding tree-based ranks. Contrary to classical learning algorithms for nonlinear model classes, the proposed algorithms are numerically stable, reliable, and require only a low level expertise of the user.

研究动机与目标

• 解决在数据有限且维度较高的统计学习中近似高维函数的挑战。
• 为基于树张量格式的非线性、秩结构化模型类开发数值稳定且用户友好的学习算法。
• 解决为函数近似选择最优树拓扑结构和基于树的秩的组合优化问题。
• 通过联合优化树结构和秩实现自适应模型选择，提升泛化能力和近似精度。

提出的方法

• 通过表示变换，将非线性树状张量格式上的风险最小化问题重新表述为一系列线性学习问题。
• 利用多线性模型结构实现稳定优化，并通过标准正则化或贪婪方法利用稀疏性。
• 采用随机树结构自适应算法，探索固定分支数的树拓扑结构，选择能最小化表示复杂度的结构。
• 通过迭代优化实现秩的自适应，支持非均匀、数据驱动的基于树的秩选择。
• 在顺序学习框架中结合树结构和秩的自适应，以最小的用户介入显著提升近似质量。
• 采用经验风险最小化与分层张量格式相结合，实现高效计算和数值稳定性。

实验结果

研究问题

• RQ1 能否利用基于树的张量格式构建自适应、数值稳定的高维函数学习算法？
• RQ2 如何同时选择最优树拓扑结构和基于树的秩，以最小化近似误差？
• RQ3 何种随机策略可有效探索高维函数近似中树结构的组合空间？
• RQ4 与固定或均匀秩模型相比，树结构与秩的联合自适应如何提升泛化能力？
• RQ5 所提出的方法能否在训练数据有限且无需专家级调优的情况下实现高精度近似？

主要发现

• 在15维测试函数上，该算法使用364个参数实现了$1.18 \times 10^{-4}$的风险，显著优于均匀秩模型。
• 对于同一函数，使用2427个参数实现了$2.95 \times 10^{-14}$的风险，表明随着模型复杂度增加，收敛速度呈指数级提升。
• 在数值实验中，随机树结构自适应算法即使从随机初始化出发，也能以高概率恢复最优树结构。
• 秩自适应带来的性能优于均匀秩选择，体现在相同参数数量下风险值更低。
• 该方法实现了稳定学习，且用户介入极少，因为它避免了非线性模型优化中常见的数值不稳定性。
• 该算法在训练样本量增加时表现出良好可扩展性，随着$n$增大，近似质量提升，尽管计算成本也随之增加。