[Paper Review] Node-By-Node Greedy Deep Learning for Interpretable Features

Ke Wu, Malik Magdon‐Ismail|arXiv (Cornell University)|Feb 19, 2016

Generative Adversarial Networks and Image Synthesis31 references3 citations

TL;DR

This paper proposes a novel node-by-node greedy deep learning algorithm that trains each neuron sequentially rather than training entire layers at once, significantly improving training speed and feature interpretability while maintaining comparable test accuracy to standard deep learning methods. The approach mimics human learning by incrementally building features using subsets of data, resulting in more interpretable internal representations without sacrificing performance.

ABSTRACT

Multilayer networks have seen a resurgence under the umbrella of deep learning. Current deep learning algorithms train the layers of the network sequentially, improving algorithmic performance as well as providing some regularization. We present a new training algorithm for deep networks which trains \emph{each node in the network} sequentially. Our algorithm is orders of magnitude faster, creates more interpretable internal representations at the node level, while not sacrificing on the ultimate out-of-sample performance.

Motivation & Objective

• To develop a faster and more interpretable alternative to standard layer-by-layer deep learning training.
• To improve feature interpretability by training each internal node sequentially, using data subsets to learn individual features.
• To maintain high out-of-sample performance comparable to standard deep learning methods.
• To explore a human-like learning process where features are built incrementally from partial data.
• To introduce 'amnesia' as a mechanism to control feature coordination and reduce interference between learned nodes.

Proposed method

• The algorithm trains each node in the network one at a time, using a greedy, sequential approach to optimize weights feeding into a single node.
• For unsupervised learning, the greedy node-by-node (GN) algorithm trains each node using a subset of data to learn a distinct feature representation.
• For supervised learning, the greedy class-by-node (GCN) algorithm trains nodes sequentially, with data partitioned by class to enhance feature specificity.
• A novel 'amnesia' mechanism is introduced to control interference between features by selectively forgetting past information during training.
• The method uses stochastic gradient descent (SGD) with a learning rate tuned per node or layer, and data is partitioned to avoid redundancy and improve feature distinctiveness.
• The algorithm is designed to be computationally efficient by reducing the number of data points used per weight update, enabling faster convergence.

Experimental results

Research questions

• RQ1 Can training deep networks node-by-node improve training speed without sacrificing predictive performance?
• RQ2 Can sequential node training produce more interpretable internal representations than simultaneous layer training?
• RQ3 How does the proposed amnesia mechanism affect feature coordination and model generalization?
• RQ4 Does the node-by-node approach better capture human-like learning patterns in feature extraction?
• RQ5 Can the algorithm scale effectively in distributed or parallel computing environments despite its sequential nature?

Key findings

• The proposed node-by-node greedy algorithm achieves training speeds that are orders of magnitude faster than standard layer-by-layer pre-training.
• The internal feature representations learned by the algorithm are significantly more interpretable, as demonstrated by clearer visualizations of learned features in the first hidden layer.
• On multiple benchmark datasets including USPS, MUSK, ISOLET, and CANCER, the algorithm achieved test accuracy comparable to standard layer-by-layer methods.
• The unsupervised GN and supervised GCN variants both produced more human-recognizable features than standard layer-by-layer approaches, especially in visualizing digit-like patterns in the USPS dataset.
• The amnesia mechanism effectively reduced interference between features, improving feature distinctiveness and model coordination.
• The algorithm demonstrated robustness across diverse data types, including high-dimensional sparse data (e.g., CNAE-9) and low-dimensional data (e.g., CANCER), with consistent performance gains in speed and interpretability.