[Paper Review] Is Extreme Learning Machine Feasible? A Theoretical Assessment (Part II)

Shaobo Lin, Xia Liu+2 more|arXiv (Cornell University)|Jan 24, 2014

Machine Learning and ELM4 citations

TL;DR

This paper provides a theoretical assessment of Extreme Learning Machine (ELM) feasibility, showing that while ELM reduces computational cost by randomizing hidden layer weights, it suffers from two key drawbacks: (1) inherent uncertainty in approximation and learning due to randomness, and (2) generalization degradation when using Gaussian-type activation functions. The study proves that ELM with Gaussian kernels performs worse than standard FNNs, but this can be remedied via coefficient regularization and multiple training trials, restoring optimal learning rates in expectation.

ABSTRACT

An extreme learning machine (ELM) can be regarded as a two stage feed-forward neural network (FNN) learning system which randomly assigns the connections with and within hidden neurons in the first stage and tunes the connections with output neurons in the second stage. Therefore, ELM training is essentially a linear learning problem, which significantly reduces the computational burden. Numerous applications show that such a computation burden reduction does not degrade the generalization capability. It has, however, been open that whether this is true in theory. The aim of our work is to study the theoretical feasibility of ELM by analyzing the pros and cons of ELM. In the previous part on this topic, we pointed out that via appropriate selection of the activation function, ELM does not degrade the generalization capability in the expectation sense. In this paper, we launch the study in a different direction and show that the randomness of ELM also leads to certain negative consequences. On one hand, we find that the randomness causes an additional uncertainty problem of ELM, both in approximation and learning. On the other hand, we theoretically justify that there also exists an activation function such that the corresponding ELM degrades the generalization capability. In particular, we prove that the generalization capability of ELM with Gaussian kernel is essentially worse than that of FNN with Gaussian kernel. To facilitate the use of ELM, we also provide a remedy to such a degradation. We find that the well-developed coefficient regularization technique can essentially improve the generalization capability. The obtained results reveal the essential characteristic of ELM and give theoretical guidance concerning how to use ELM.

Motivation & Objective

• To theoretically assess the feasibility of Extreme Learning Machine (ELM) beyond its empirical success.
• To investigate the negative consequences of ELM's random weight assignment in hidden layers.
• To determine whether ELM degrades generalization capability under certain activation functions.
• To propose remedies for the uncertainty and generalization degradation problems in ELM.
• To establish a theoretical foundation for selecting activation functions and regularization strategies in ELM.

Proposed method

• Analyzes ELM within the framework of statistical learning theory, focusing on generalization error and approximation error.
• Uses concentration inequalities and regularization techniques to bound the expected generalization error.
• Applies $l^2$ coefficient regularization to stabilize the output weight learning and improve generalization.
• Employs multiple training trials to mitigate the uncertainty caused by random weight initialization.
• Derives theoretical error bounds using sample-dependent hypothesis spaces and kernel-based learning models.
• Compares ELM with standard FNNs using Gaussian kernels, showing ELM's suboptimal convergence rate without regularization.

Experimental results

Research questions

• RQ1 Does the randomness in ELM’s hidden layer weights introduce an uncertainty problem in both approximation and learning?
• RQ2 Can ELM degrade generalization capability for certain activation functions, even with random weight assignment?
• RQ3 Is the generalization performance of ELM with Gaussian-type activation functions worse than that of standard FNNs?
• RQ4 Can $l^2$ regularization restore optimal learning rates in ELM, particularly for Gaussian kernels?
• RQ5 What conditions on activation functions determine whether ELM degrades or preserves generalization capability?

Key findings

• ELM with Gaussian-type activation functions degrades generalization capability compared to standard FNNs, as shown by theoretical comparison of learning rates.
• The generalization error of ELM with Gaussian kernels is fundamentally worse than that of FNNs under the same conditions.
• Multiple training trials effectively mitigate the uncertainty problem in ELM, reducing the risk of failure in single runs.
• Applying $l^2$ coefficient regularization to output weights enables ELM to achieve almost optimal learning rates in expectation, matching FNN performance.
• The theoretical error bound for regularized ELM scales as $O(m^{-\frac{2r}{2r+d} + \varepsilon})$, approaching the optimal rate of FNNs when regularization is properly tuned.
• A general criterion for classifying activation functions based on their impact on ELM’s generalization capability remains an open question, though the paper provides a foundation for future work.