[Paper Review] Online and Differentially-Private Tensor Decomposition

Yining Wang, Animashree Anandkumar|arXiv (Cornell University)|Jun 20, 2016

Tensor decomposition and applications13 citations

TL;DR

This paper proposes the first online and differentially private tensor decomposition algorithm using a noise-calibrated tensor power method, achieving linear memory usage and strong privacy guarantees. It provides the first theoretical analysis showing that privacy parameters do not polynomially depend on tensor dimension under incoherence assumptions, significantly improving noise tolerance over prior work.

ABSTRACT

In this paper, we resolve many of the key algorithmic questions regarding robustness, memory efficiency, and differential privacy of tensor decomposition. We propose simple variants of the tensor power method which enjoy these strong properties. We present the first guarantees for online tensor power method which has a linear memory requirement. Moreover, we present a noise calibrated tensor power method with efficient privacy guarantees. At the heart of all these guarantees lies a careful perturbation analysis derived in this paper which improves up on the existing results significantly.

Motivation & Objective

• Address the lack of robust, memory-efficient, and private tensor decomposition algorithms for large-scale data.
• Enable online tensor decomposition with linear memory usage by avoiding explicit tensor construction.
• Provide the first differential privacy guarantees for tensor decomposition with utility preservation.
• Improve noise tolerance in tensor power methods through refined perturbation analysis.
• Ensure privacy parameters are independent of tensor dimension under natural incoherence assumptions.

Proposed method

• Propose an online tensor power method that processes data streams without forming the full tensor, reducing memory to O(d) for a d-dimensional tensor.
• Introduce a noise-calibrated tensor power method that injects noise in a way that preserves privacy while maintaining recovery accuracy.
• Derive a novel perturbation analysis that significantly improves noise tolerance bounds compared to prior work.
• Use tensor incoherence assumptions to show that privacy parameters have no polynomial dependence on dimension d.
• Leverage online stochastic updates to compute tensor components incrementally from i.i.d. data samples.
• Apply Wedin’s theorem and matrix perturbation bounds to analyze component recovery under noise.

Experimental results

Research questions

• RQ1 Can tensor decomposition be performed efficiently in an online, streaming setting with linear memory?
• RQ2 Can differential privacy be achieved in tensor decomposition without sacrificing utility or introducing polynomial dependence on dimension?
• RQ3 How much noise can the tensor power method tolerate while still recovering components accurately?
• RQ4 What perturbation bounds are necessary and sufficient for stable tensor component recovery under noise?
• RQ5 Can privacy guarantees be maintained without requiring full tensor storage or exponential computation?

Key findings

• The online tensor power method achieves O(dk) memory usage and Õ(nk²d) time complexity for a rank-k tensor with n samples.
• The proposed noise-calibrated method provides the first differential privacy guarantees for tensor decomposition with privacy parameters independent of tensor dimension d under incoherence.
• The perturbation analysis improves noise tolerance bounds, allowing recovery under significantly higher noise levels than previous methods.
• The method ensures correct recovery of tensor components with high probability under mild assumptions on tensor incoherence and noise distribution.
• Sample complexity analysis shows efficient convergence with Õ(nk²d) time and O(dk) memory, suitable for large-scale streaming data.
• Theoretical guarantees hold even when noise is added via differential privacy, ensuring both privacy and utility.