[Paper Review] Learning Sparse Deep Feedforward Networks via Tree Skeleton Expansion

Zhourong Chen, Xiaopeng Li+1 more|arXiv (Cornell University)|Mar 16, 2018

Bayesian Modeling and Causal Inference23 references3 citations

TL;DR

This paper proposes Tree Skeleton Expansion Networks (TSE-Nets), a novel unsupervised method for learning sparse, structured deep feedforward networks by leveraging hierarchical latent tree models (HLTMs) to identify correlated feature groups. By expanding connections from a tree skeleton derived via hierarchical latent tree analysis (HLTA), TSE-Nets achieve comparable or better classification accuracy than manually tuned networks with significantly fewer parameters and improved interpretability.

ABSTRACT

Despite the popularity of deep learning, structure learning for deep models remains a relatively under-explored area. In contrast, structure learning has been studied extensively for probabilistic graphical models (PGMs). In particular, an efficient algorithm has been developed for learning a class of tree-structured PGMs called hierarchical latent tree models (HLTMs), where there is a layer of observed variables at the bottom and multiple layers of latent variables on top. In this paper, we propose a simple method for learning the structures of feedforward neural networks (FNNs) based on HLTMs. The idea is to expand the connections in the tree skeletons from HLTMs and to use the resulting structures for FNNs. An important characteristic of FNN structures learned this way is that they are sparse. We present extensive empirical results to show that, compared with standard FNNs tuned-manually, sparse FNNs learned by our method achieve better or comparable classification performance with much fewer parameters. They are also more interpretable.

Motivation & Objective

• To address the challenge of manual architecture design in deep feedforward networks (FNNs), which is time-consuming and often suboptimal.
• To develop an unsupervised method for learning sparse, structured FNNs that are both efficient and interpretable.
• To leverage probabilistic graphical models (PGMs) and hierarchical latent tree analysis (HLTA) to discover underlying feature correlations for network structure learning.
• To improve model interpretability by learning feature groupings that align with meaningful semantic or spatial structures in data.
• To demonstrate that structure learning via tree skeleton expansion yields models that outperform both standard FNNs and pruned FNNs in accuracy, sparsity, and interpretability.

Proposed method

• Apply hierarchical latent tree analysis (HLTA) to partition input features into strongly correlated groups, modeling each group with a latent variable to form a tree-structured PGM.
• Construct a tree skeleton from the resulting hierarchical latent tree model (HLTMs), representing the core connectivity pattern of the FNN.
• Expand the tree skeleton by adding additional edges to capture salient probabilistic dependencies not represented in the initial skeleton.
• Use the expanded PGM core as the backbone of the FNN, integrating additional hidden units (layer h₃) and output units to enable direct feature contribution from all layers.
• Design the network with two types of h₃ neurons: those connected to the top of the PGM core (forming the backbone) and those providing narrow skip-paths from lower layers to the output.
• Train the final TSE-Net model end-to-end without pre-training or weight-based pruning, relying solely on the structure learned from data correlations.

Experimental results

Research questions

• RQ1 Can unsupervised structure learning based on hierarchical latent tree models produce sparse FNNs that match or exceed the performance of manually designed FNNs?
• RQ2 How does the interpretability of hidden units in TSE-Nets compare to that of standard FNNs and pruned FNNs, especially in terms of semantic coherence of learned features?
• RQ3 To what extent can the tree skeleton expansion method capture spatial or structural correlations in data (e.g., neighboring pixels in MNIST) without using spatial information?
• RQ4 How does the performance of TSE-Nets compare to weight-pruning-based methods in achieving sparse connectivity with similar parameter counts?
• RQ5 Can the proposed method generalize to diverse data types, including tabular, text, and image data, while maintaining high accuracy and low parameter count?

Key findings

• TSE-Nets achieve better or comparable classification performance to manually tuned FNNs across 12 Tox21 tasks, despite having significantly fewer parameters.
• On the MNIST dataset, the method successfully groups neighboring pixels into coherent clusters without using spatial coordinates, reflecting the structural patterns of handwritten digits.
• TSE-Nets outperform both standard FNNs and pruned FNNs in interpretability, with hidden units showing higher average cosine similarity between top-10 correlated words (e.g., 0.1632 on Yelp Review vs. 0.1117 for standard FNN).
• When compared to a weight-pruning baseline (Han et al., 2015), TSE-Nets achieve similar performance with the same number of parameters, but without requiring pre-training or iterative weight adjustment.
• The model’s backbone structure, combined with narrow skip-paths, enables effective feature propagation while maintaining sparsity and interpretability.
• The method demonstrates strong generalization across diverse data types, including text (Yelp, DBPedia), tabular (Tox21), and image (MNIST) datasets, with consistent gains in efficiency and interpretability.