[Paper Review] TensorLy: Tensor Learning in Python

Jean Kossaifi, Yannis Panagakis+2 more|University of Twente Research Information|Oct 29, 2016

Tensor decomposition and applicationsMathematics21 references210 citations

TL;DR

TensorLy is a BSD-licensed Python library providing state-of-the-art tensor learning with a flexible backend system that abstracts computations over NumPy, MXNet, PyTorch, TensorFlow, and CuPy, enabling scalable tensor operations and deep tensorized networks.

ABSTRACT

Tensors are higher-order extensions of matrices. While matrix methods form the cornerstone of machine learning and data analysis, tensor methods have been gaining increasing traction. However, software support for tensor operations is not on the same footing. In order to bridge this gap, we have developed \emph{TensorLy}, a high-level API for tensor methods and deep tensorized neural networks in Python. TensorLy aims to follow the same standards adopted by the main projects of the Python scientific community, and seamlessly integrates with them. Its BSD license makes it suitable for both academic and commercial applications. TensorLy's backend system allows users to perform computations with NumPy, MXNet, PyTorch, TensorFlow and CuPy. They can be scaled on multiple CPU or GPU machines. In addition, using the deep-learning frameworks as backend allows users to easily design and train deep tensorized neural networks. TensorLy is available at https://github.com/tensorly/tensorly

Motivation & Objective

• Introduce TensorLy as a high-level Python API for tensor methods and deep tensorized neural networks.
• Provide a backend-agnostic computation framework to switch between NumPy, MXNet, PyTorch, TensorFlow, and CuPy.
• Offer state-of-the-art tensor operations, decompositions, and regression methods with solid testing and documentation.
• Demonstrate performance and scalability advantages and accessibility for integration with the Python scientific stack.

Proposed method

• Implement tensor algebra operations and decompositions (CP, Tucker, nonnegative variants, Robust Tensor PCA, low-rank tensor regression).
• Provide a flexible backend system to execute computations on NumPy, MXNet, PyTorch, CuPy, and TensorFlow with eager execution support.
• Integrate with deep learning frameworks to enable deep tensorized neural networks.
• Ensure code quality with extensive unit tests and documentation (97% coverage at writing).
• Advertise an efficient unfolding strategy for mode-n tensors to improve performance on CPU/GPU.

Experimental results

Research questions

• RQ1 Can TensorLy provide a comprehensive, open-source tensor learning library with broad backend support for Python data science workflows?
• RQ2 How does TensorLy perform tensor decompositions and learning operations across different backends and hardware?
• RQ3 What are the design choices that enable easy integration of tensor methods with deep learning frameworks?
• RQ4 What is the comparative performance of TensorLy's tensor operations relative to existing tools on CPU and GPU?
• RQ5 Can TensorLy scale to multi-device setups while remaining easy to use for researchers?

Key findings

• TensorLy offers state-of-the-art tensor learning methods including CP and Tucker decompositions and their variants.
• The library provides a flexible backend system that can use NumPy, MXNet, PyTorch, TensorFlow, or CuPy for computations.
• TensorLy is BSD-licensed and open source with documentation and extensive unit tests (97% coverage).
• Experiments show competitive speed for tensor decompositions on CPU and GPU across backends.
• TensorLy enables combining tensor methods with deep learning and supports scalable execution on multi-CPU and multi-GPU setups.
• Future work includes adding more methods (PARAFAC2, DEDICOM, tensor networks) and exploring BLAS primitive extensions for performance.