[Paper Review] An algorithm for online tensor prediction

John Pothier, Josh Girson+1 more|arXiv (Cornell University)|Jul 28, 2015

Tensor decomposition and applications13 references3 citations

TL;DR

This paper proposes an online tensor prediction algorithm using tensor exponentiated gradient descent based on tensor-SVD and von Neumann divergence generalizations, achieving provable regret bounds for 3D tensor completion under sequential, non-adaptive observations. The method enables low-rank tensor learning with theoretical performance guarantees, outperforming existing online convex tensor completion methods on synthetic social network rating data.

ABSTRACT

We present a new method for online prediction and learning of tensors ($N$-way arrays, $N >2$) from sequential measurements. We focus on the specific case of 3-D tensors and exploit a recently developed framework of structured tensor decompositions proposed in [1]. In this framework it is possible to treat 3-D tensors as linear operators and appropriately generalize notions of rank and positive definiteness to tensors in a natural way. Using these notions we propose a generalization of the matrix exponentiated gradient descent algorithm [2] to a tensor exponentiated gradient descent algorithm using an extension of the notion of von-Neumann divergence to tensors. Then following a similar construction as in [3], we exploit this algorithm to propose an online algorithm for learning and prediction of tensors with provable regret guarantees. Simulations results are presented on semi-synthetic data sets of ratings evolving in time under local influence over a social network. The result indicate superior performance compared to other (online) convex tensor completion methods.

Motivation & Objective

• To address online prediction of 3D tensors from sequential, non-adaptive measurements with performance guarantees.
• To extend matrix exponentiated gradient descent to tensors using structured tensor decompositions and generalized divergence notions.
• To provide a framework for online learning of low-rank tensors under convex loss functions with bounded Lipschitz continuity.
• To establish theoretical regret bounds for online tensor prediction using tensor-SVD and block-diagonal matrix representations.
• To demonstrate superior performance on semi-synthetic social network rating data compared to existing online convex tensor completion methods.

Proposed method

• Proposes a tensor exponentiated gradient descent (TEG) algorithm by generalizing the matrix exponentiated gradient method to tensors using tensor-SVD and block-diagonal matrix representations.
• Defines a tensor-specific von Neumann divergence using the tensor-SVD framework, enabling convex optimization over tensor spaces.
• Represents each tensor as a linear operator on group-rings over cyclic groups, allowing structured low-rank approximation.
• Uses a linear approximation of the loss function to bound regret, reducing the non-linear online tensor problem to a linear one.
• Applies Corollary VII.1 to derive a regret bound dependent on the trace of squared loss operators and tensor rank parameters.
• Employs a step size rule based on the inverse of the square root of time and tensor complexity parameters to minimize regret.

Experimental results

Research questions

• RQ1 Can a provably convergent online algorithm be designed for tensor prediction under sequential, non-adaptive observations?
• RQ2 How can the matrix exponentiated gradient method be generalized to tensors using structured tensor decompositions?
• RQ3 What is the appropriate generalization of von Neumann divergence and rank to 3D tensors for optimization purposes?
• RQ4 Can the regret of online tensor prediction be bounded in terms of tensor rank and loss function properties?
• RQ5 How does the proposed method compare to existing online convex tensor completion techniques in practice?

Key findings

• The proposed Online Tensor Exponentiated Gradient (OTEG) algorithm achieves a regret bound of $ 2Gig(Teta(k)ig)^{1/2} $, where $ G $ is the gradient bound and $ T $ is time.
• The regret bound scales with the square root of time and the product of tensor rank and loss complexity parameters, indicating sublinear growth.
• The algorithm outperforms other online convex tensor completion methods on semi-synthetic social network rating data with local influence patterns.
• Theoretical guarantees hold under the condition $ \eta \|\text{blkdiag}(\widehat{\bm{\mathscr{L}}}_t)\| \leq 1 $, which is satisfied for sufficiently large $ T $.
• For small $ T $, the regret remains bounded by $ 2GT $, which is still within the derived bound due to the normalization of $ \|\beta\|_1 \geq 1 $.
• The method leverages tensor-SVD to represent tensors as linear operators on group-rings, enabling efficient and structured low-rank learning.