[论文解读] Tensor Networks Meet Neural Networks: A Survey and Future Perspectives

Maolin Wang, Pan Yu+5 more|arXiv (Cornell University)|Jan 22, 2023

Computational Physics and Python Applications被引用 14

一句话总结

本综述回顾如何将 tensor networks (TNs) 和 neural networks (NNs) 整合到 tensorial neural networks (TNNs)，聚焦于网络压缩、信息融合和量子电路仿真，并附带工具和未来方向。

ABSTRACT

Tensor networks (TNs) and neural networks (NNs) are two fundamental data modeling approaches. TNs were introduced to solve the curse of dimensionality in large-scale tensors by converting an exponential number of dimensions to polynomial complexity. As a result, they have attracted significant attention in the fields of quantum physics and machine learning. Meanwhile, NNs have displayed exceptional performance in various applications, e.g., computer vision, natural language processing, and robotics research. Interestingly, although these two types of networks originate from different observations, they are inherently linked through the typical multilinearity structure underlying both TNs and NNs, thereby motivating a significant number of developments regarding combinations of TNs and NNs. In this paper, we refer to these combinations as tensorial neural networks~(TNNs) and present an introduction to TNNs from both data processing and model architecture perspectives. From the data perspective, we explore the capabilities of TNNs in multi-source fusion, multimodal pooling, data compression, multi-task training, and quantum data processing. From the model perspective, we examine TNNs' integration with various architectures, including Convolutional Neural Networks, Recurrent Neural Networks, Graph Neural Networks, Transformers, Large Language Models, and Quantum Neural Networks. Furthermore, this survey also explores methods for improving TNNs, examines flexible toolboxes for implementing TNNs, and documents TNN development while highlighting potential future directions. To the best of our knowledge, this is the first comprehensive survey that bridges the connections among NNs and TNs. We provide a curated list of TNNs at https://github.com/tnbar/awesome-tensorial-neural-networks.

研究动机与目标

• 推动将 TNs 和 NNs 融合以克服维度和效率挑战。
• 将 tensorial neural networks (TNNs) 作为连接 TNs、NNs 与量子电路的统一框架引入。
• 综述三个核心应用：网络压缩、信息融合和量子电路仿真。
• 就训练策略、实现技术和可用工具箱提供指导。

提出的方法

• 用图示表示解释张量网络基础及常见格式（CP、Tucker、TT/MPS、TR、HT、PEPS）及其示意表示。
• 展示通过分解权重张量来实现紧凑的 NN 架构（网络压缩）。
• 通过张量融合层和多模态池化展示信息融合以捕捉高阶交互。
• 讨论将 TN 作为量子电路模拟器来连接经典 NN 和量子电路（tensorial NN 概念）。
• 概述训练与实现的考虑因素，包括秩选择与硬件加速，并总结可用的工具箱。

实验结果

研究问题

• RQ1 TNs 在不损失性能的前提下压缩神经网络的主要方式有哪些？
• RQ2 张量网络如何促进神经网络架构中多模态数据的信息融合？
• RQ3 TN 在何种程度上能够模拟或桥接到量子电路，从而实现 tensorial neural networks？
• RQ4 构建和部署 TNN 的实际训练策略与工具链有哪些？
• RQ5 在压缩、融合和量子仿真等方面，未来 TNN 发展有哪些有前景的方向？

主要发现

• TNNs 提供紧凑的表示，能够在 CNNs、RNNs、Transformers、GNNs 和 RBMs 中减少参数数量与计算成本。
• 基于张量的融合单元（如张量融合层、MUTAN）在参数效率下捕获更高阶的多模态交互。
• 张量网络可作为有效的量子电路模拟器，在可扩展的量子硬件可用之前，便于探索量子神经网络。
• 存在实用的训练策略（如秩选择、稳定训练、混合精度）以及若干硬件加速和工具箱选项，以支持 TNN 的开发。
• 该综述记录了广泛的 TNN 变体及应用，并提供了一个精选的张量神经网络资源库，以指导未来工作。