[Paper Review] Guaranteed Scalable Learning of Latent Tree Models

Furong Huang, U. N. Niranjan+4 more|CaltechAUTHORS (California Institute of Technology)|Jun 18, 2014

Bayesian Modeling and Causal Inference16 references4 citations

TL;DR

This paper presents a guaranteed, scalable method for learning latent tree models by integrating structure and parameter estimation via a divide-and-conquer framework. It uses tensor decomposition and method of moments for consistent parameter recovery with log(p) computational complexity and O(log p) parallel time, achieving sample complexity log(k) for both structure and parameter estimation across discrete, Gaussian, and Gaussian mixture models.

ABSTRACT

We present an integrated approach for structure and parameter estimation in latent tree graphical models. Our overall approach follows a "divide-and-conquer" strategy that learns models over small groups of variables and iteratively merges onto a global solution. The structure learning involves combinatorial operations such as minimum spanning tree construction and local recursive grouping; the parameter learning is based on the method of moments and on tensor decompositions. Our method is guaranteed to correctly recover the unknown tree structure and the model parameters with low sample complexity for the class of linear multivariate latent tree models which includes discrete and Gaussian distributions, and Gaussian mixtures. Our bulk asynchronous parallel algorithm is implemented in parallel and the parallel computation complexity increases only logarithmically with the number of variables and linearly with dimensionality of each variable.

Motivation & Objective

• To address the lack of consistent, scalable, and parallelizable structure and parameter estimation in latent tree models.
• To overcome the sequential dependency between structure learning and parameter estimation in existing methods.
• To achieve global consistency in structure and parameter recovery with low sample complexity.
• To enable efficient, bulk-asynchronous parallel computation with logarithmic time complexity in the number of variables.
• To extend tensor decomposition methods to hierarchical models with guaranteed alignment and recovery of latent parameters.

Proposed method

• The method employs a divide-and-conquer strategy, recursively grouping variables into local sub-trees for parallel processing.
• Structure learning is performed via minimum spanning tree (MST) construction on pairwise multivariate information distances.
• Local recursive grouping (LRG) identifies sub-trees and enables scalable, distributed structure estimation.
• Parameter estimation uses the method of moments and higher-order tensor decompositions to recover latent parameters consistently.
• A novel tensor spectral alignment procedure aligns parameters across sub-trees using permutation matrices derived from singular value decomposition (SVD) and matrix alignment techniques.
• The algorithm leverages sparse matrix operations and randomized SVD to reduce computational cost, especially for high-dimensional and sparse data.

Experimental results

Research questions

• RQ1 Can latent tree structure and parameters be recovered consistently with sub-polynomial computational complexity?
• RQ2 Can tensor decomposition methods be extended to hierarchical latent tree models with guaranteed alignment and recovery?
• RQ3 Is it possible to achieve parallel, scalable learning of latent tree models without sequential dependency between structure and parameter estimation?
• RQ4 What is the minimal sample complexity required for consistent recovery of both structure and parameters in linear multivariate latent tree models?
• RQ5 How can sparse and high-dimensional data be efficiently processed in latent tree learning with theoretical guarantees?

Key findings

• The method achieves global consistency in structure and parameter estimation with computational complexity O(log p), significantly improving over the state-of-the-art O(poly(p)) complexity.
• The sample complexity for consistent structure recovery is O(log k), and for parameter recovery via tensor decomposition is O(log k), both with high probability.
• The algorithm guarantees consistent parameter estimation using the method of moments and tensor decomposition, the first such guarantee for latent tree models.
• The parallel computation complexity increases only logarithmically with the number of variables and linearly with dimensionality, enabling efficient large-scale learning.
• The use of sparse matrix embeddings and randomized SVD reduces time and memory costs, especially for high-dimensional and sparse data.
• The framework supports user-guided structure refinement via neighborhood selection and BIC scoring, with dynamic parameter re-estimation without re-running EM.