[Paper Review] Distributed Flexible Nonlinear Tensor Factorization

Shandian Zhe, Kai Zhang+5 more|ArXiv.org|Apr 27, 2016

Tensor decomposition and applications24 references3 citations

TL;DR

This paper proposes a distributed, flexible nonlinear tensor factorization model that avoids computationally expensive Kronecker-product structures by using Gaussian process priors with kernel-based covariance, enabling arbitrary selection of meaningful tensor entries for training. It introduces a tight variational evidence lower bound (ELBO) that enables fully decoupled, parallel inference in a key-value-free MapReduce framework, achieving state-of-the-art predictive performance and near-linear scalability on large-scale sparse tensors, with a 20% improvement in CTR prediction over logistic regression and SVM.

ABSTRACT

Tensor factorization is a powerful tool to analyse multi-way data. Compared with traditional multi-linear methods, nonlinear tensor factorization models are capable of capturing more complex relationships in the data. However, they are computationally expensive and may suffer severe learning bias in case of extreme data sparsity. To overcome these limitations, in this paper we propose a distributed, flexible nonlinear tensor factorization model. Our model can effectively avoid the expensive computations and structural restrictions of the Kronecker-product in existing TGP formulations, allowing an arbitrary subset of tensorial entries to be selected to contribute to the training. At the same time, we derive a tractable and tight variational evidence lower bound (ELBO) that enables highly decoupled, parallel computations and high-quality inference. Based on the new bound, we develop a distributed inference algorithm in the MapReduce framework, which is key-value-free and can fully exploit the memory cache mechanism in fast MapReduce systems such as SPARK. Experimental results fully demonstrate the advantages of our method over several state-of-the-art approaches, in terms of both predictive performance and computational efficiency. Moreover, our approach shows a promising potential in the application of Click-Through-Rate (CTR) prediction for online advertising.

Motivation & Objective

• To address the computational inefficiency and learning bias in existing nonlinear tensor factorization models due to full-tensor modeling and Kronecker-product structures.
• To enable flexible training by allowing arbitrary subsets of tensor entries—especially meaningful non-zero and balanced zero entries—to contribute to learning.
• To develop a tractable, tight variational evidence lower bound (ELBO) that supports highly parallel, decoupled inference without sequential E-M updates.
• To design a key-value-free distributed inference algorithm in MapReduce that leverages memory caching and avoids costly data shuffling.
• To demonstrate superior performance on both predictive accuracy and computational efficiency, especially in large-scale, sparse tensor applications like CTR prediction.

Proposed method

• Proposes a nonlinear tensor factorization model using Gaussian process priors over tensor entries, with inputs formed by concatenating latent factors from each mode.
• Employs a kernel function to model complex, nonlinear interactions without requiring Kronecker-product covariance structures.
• Derives a tight variational ELBO using functional derivatives and convex conjugates, subsuming optimal variational posteriors and enabling efficient, parallel optimization.
• Develops a distributed inference algorithm in the MapReduce framework that computes and transmits only global gradient vectors, eliminating key-value pairs and reducing I/O overhead.
• Utilizes the SPARK system to fully exploit memory cache mechanisms, avoiding disk I/O bottlenecks from data shuffling.
• Applies the model to CTR prediction by constructing a four-mode tensor from user, ad, publisher, and page-section interactions, with balanced sampling of clicks and non-clicks.

Experimental results

Research questions

• RQ1 Can a nonlinear tensor factorization model avoid the computational burden of Kronecker-product covariance while maintaining high predictive accuracy?
• RQ2 How can a flexible training mechanism be designed to include only meaningful tensor entries and reduce learning bias in sparse data?
• RQ3 Can a tight and tractable variational lower bound be derived to enable fully parallel, decoupled inference without sequential E-M updates?
• RQ4 To what extent can a key-value-free MapReduce algorithm improve scalability and reduce I/O overhead in distributed tensor factorization?
• RQ5 Does the proposed method achieve superior performance on real-world sparse tensor tasks such as CTR prediction compared to state-of-the-art approaches?

Key findings

• The proposed model outperforms GigaTensor, DinTucker, and InfTuckerEx in predictive accuracy on all three benchmark datasets (ACC, DBLP, NELL), with a 20.7% average AUC improvement on CTR prediction.
• On the DBLP dataset, the model achieved an average per-iteration runtime of 1.45 minutes, significantly faster than GigaTensor (15.4 min) and DinTucker (20.5 min).
• The model demonstrated near-linear scalability across computational nodes, with substantial speedups due to key-value-free gradient transmission and memory cache utilization in SPARK.
• The model achieved a 20% improvement in AUC over logistic regression and linear SVM on CTR prediction, even without incorporating side features like user profiles or ad attributes.
• The tight ELBO derivation enabled high-quality inference without sequential E-M updates, supporting efficient, parallel optimization across distributed nodes.
• The method effectively mitigates learning bias in extreme data sparsity by allowing selective inclusion of non-zero and balanced zero entries in training.