[Paper Review] Automatic Construction of Multi-layer Perceptron Network from Streaming Examples

Mahardhika Pratama, Choiru Za’in+3 more|arXiv (Cornell University)|Oct 8, 2019

Data Stream Mining Techniques26 references4 citations

TL;DR

This paper proposes NADINE, a self-organizing deep neural network that autonomously constructs and evolves a multi-layer perceptron (MLP) structure from streaming data in real time. It dynamically adjusts depth and width using bias-variance estimation, mitigates catastrophic forgetting via adaptive memory and soft forgetting, and achieves superior performance across nine data stream classification and regression tasks without user-defined thresholds.

ABSTRACT

Autonomous construction of deep neural network (DNNs) is desired for data streams because it potentially offers two advantages: proper model's capacity and quick reaction to drift and shift. While the self-organizing mechanism of DNNs remains an open issue, this task is even more challenging to be developed for standard multi-layer DNNs than that using the different-depth structures, because the addition of a new layer results in information loss of previously trained knowledge. A Neural Network with Dynamically Evolved Capacity (NADINE) is proposed in this paper. NADINE features a fully open structure where its network structure, depth and width, can be automatically evolved from scratch in an online manner and without the use of problem-specific thresholds. NADINE is structured under a standard MLP architecture and the catastrophic forgetting issue during the hidden layer addition phase is resolved using the proposal of soft-forgetting and adaptive memory methods. The advantage of NADINE, namely elastic structure and online learning trait, is numerically validated using nine data stream classification and regression problems where it demonstrates performance improvement over prominent algorithms in all problems. In addition, it is capable of dealing with data stream regression and classification problems equally well.

Motivation & Objective

• To address the challenge of autonomous, incremental deep learning in data streams with evolving network structures.
• To resolve catastrophic forgetting during structural evolution in standard MLPs, particularly during hidden layer addition.
• To eliminate reliance on problem-specific thresholds or fixed network architectures in online learning.
• To enable end-to-end online learning for both classification and regression tasks in non-stationary data streams.
• To develop a fully open, reconfigurable DNN architecture that evolves in real time with minimal human intervention.

Proposed method

• NADINE employs a Network Significance (NS) method to estimate bias and variance at the hidden unit level, guiding dynamic growth and pruning of units.
• It uses a drift detection mechanism to trigger structural changes, enabling on-demand expansion or contraction of the network depth.
• The adaptive memory mechanism preserves relevance of previously learned knowledge by tracking the importance of each layer.
• Soft forgetting adjusts learning sensitivity per layer during SGD based on layer relevance, reducing catastrophic forgetting during structural updates.
• The method operates in a one-pass, online manner, processing each example only once, ensuring scalability for streaming data.
• Structural decisions (addition, pruning, depth change) are made without predefined thresholds, relying on statistical estimation of generalization error.

Experimental results

Research questions

• RQ1 Can a standard MLP architecture be autonomously evolved in depth and width during online learning without catastrophic forgetting?
• RQ2 How can catastrophic forgetting be effectively mitigated during incremental structural changes in a standard MLP?
• RQ3 Can a self-organizing DNN maintain high performance across both classification and regression tasks in data streams?
• RQ4 To what extent does the absence of problem-specific thresholds improve generalization and adaptability in streaming environments?
• RQ5 How do adaptive memory and soft forgetting contribute to preserving knowledge during structural evolution?

Key findings

• NADINE outperforms prominent online learning algorithms across all nine data stream classification and regression problems in the prequential test-then-train protocol.
• The ablation study shows that removing the layer-growing mechanism reduces accuracy to 85.85±10.41, indicating its critical role in performance.
• The soft forgetting mechanism reduces performance degradation during structural changes, with accuracy dropping to 86.23±11.63 when disabled.
• The adaptive memory mechanism contributes significantly, as disabling it leads to a 1.11% drop in accuracy (from 89.35±6.79 to 88.24±7.32) on average.
• NADINE achieves a mean accuracy of 89.06±6.42 on the HEPMASS dataset, demonstrating robustness in complex, high-dimensional data streams.
• The method successfully handles both classification and regression tasks equally well, with no performance degradation in either modality.