[Paper Review] Discovery of Latent Factors in High-dimensional Data Using Tensor Methods

Furong Huang|arXiv (Cornell University)|Jun 10, 2016

Tensor decomposition and applications133 references3 citations

TL;DR

This dissertation proposes tensor decomposition methods to discover latent factors in high-dimensional data, leveraging spectral techniques for unsupervised learning. It demonstrates global convergence and state-of-the-art performance in learning latent tree models, community detection, and neuroscience applications like cell type discovery from spatial gene expression data.

ABSTRACT

Unsupervised learning aims at the discovery of hidden structure that drives the observations in the real world. It is essential for success in modern machine learning. Latent variable models are versatile in unsupervised learning and have applications in almost every domain. Training latent variable models is challenging due to the non-convexity of the likelihood objective. An alternative method is based on the spectral decomposition of low order moment tensors. This versatile framework is guaranteed to estimate the correct model consistently. My thesis spans both theoretical analysis of tensor decomposition framework and practical implementation of various applications. This thesis presents theoretical results on convergence to globally optimal solution of tensor decomposition using the stochastic gradient descent, despite non-convexity of the objective. This is the first work that gives global convergence guarantees for the stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points. This thesis also presents large-scale deployment of spectral methods carried out on various platforms. Dimensionality reduction techniques such as random projection are incorporated for a highly parallel and scalable tensor decomposition algorithm. We obtain a gain in both accuracies and in running times by several orders of magnitude compared to the state-of-art variational methods. To solve real world problems, more advanced models and learning algorithms are proposed. This thesis discusses generalization of LDA model to mixed membership stochastic block model for learning user communities in social network, convolutional dictionary model for learning word-sequence embeddings, hierarchical tensor decomposition and latent tree structure model for learning disease hierarchy, and spatial point process mixture model for detecting cell types in neuroscience.

Motivation & Objective

• Address the challenge of uncovering hidden structures in high-dimensional data where traditional methods fail due to complexity and scalability.
• Develop scalable and provably convergent tensor-based algorithms for unsupervised learning in complex models such as latent tree models and community detection.
• Enable accurate discovery of cell types and gene expression profiles in neuroscience using spatial point process mixture models and tensor decomposition.
• Extend spectral methods beyond standard matrix factorization to handle hierarchical and invariant structures in data.
• Provide a unified framework for learning models with group invariance, such as rotation or scaling, using tensor decomposition.

Proposed method

• Utilizes tensor decomposition as a core framework to extract latent factors from high-dimensional data, enabling global optimization and convergence guarantees.
• Applies spectral methods to latent tree models and multi-view models, using higher-order moments to identify underlying structures.
• Employs tensor sketching and distributed computation to reduce computational cost and scale to large datasets.
• Integrates variational inference with tensor decomposition in a spatial point process mixture model for neuroscience data.
• Leverages symmetric and gene-specific Dirichlet priors to improve biological fidelity in modeling cell-type proportions and gene expression.
• Develops efficient C++ and CUDA implementations for tensor-based topic modeling and community detection, enabling real-world deployment.

Experimental results

Research questions

• RQ1 Can tensor decomposition methods reliably and efficiently uncover latent factors in high-dimensional, complex data with provable convergence?
• RQ2 How can tensor-based methods improve the accuracy of latent variable model learning compared to existing approaches in healthcare and neuroscience?
• RQ3 To what extent can tensor methods handle hierarchical and invariant structures in data, such as those found in graphical models or neural networks?
• RQ4 Can tensor decomposition be used to model spatial and gene expression patterns in brain tissue to discover novel cell types?
• RQ5 What are the computational and statistical trade-offs in using tensor sketching and distributed frameworks for large-scale tensor learning?

Key findings

• The proposed tensor decomposition framework achieves global convergence and outperforms state-of-the-art methods in learning latent tree models and community detection.
• In neuroscience, the model improves prediction accuracy of gene expression profiles when compared to single-cell RNA-seq data, demonstrating superior performance over existing methods.
• The use of non-uniform Dirichlet priors for cell-type proportions and gene-specific priors enhances biological realism and model fidelity.
• Distributed and sketch-based tensor algorithms significantly reduce computational cost while maintaining accuracy, enabling scalable deployment on large datasets.
• The framework successfully discovers distinct neuronal cell types and their spatial distributions using in situ hybridization data from the Allen Brain Atlas.
• Tensor-based methods are shown to be effective for a wide range of applications, including text, image, social networks, and healthcare analytics, confirming their versatility and robustness.