[Paper Review] Constrained Extreme Learning Machines: A Study on Classification Cases

Wentao Zhu, Jun Miao+1 more|arXiv (Cornell University)|Jan 25, 2015

Machine Learning and ELM54 references16 citations

TL;DR

This paper proposes Constrained Extreme Learning Machines (CELMs), a novel approach that improves generalization in classification by selecting hidden neurons from a constrained vector space derived from sample distributions, rather than using completely random selection as in traditional ELM. The method achieves better accuracy than standard ELM, SVM, and related methods while maintaining ELM’s fast learning speed.

ABSTRACT

Extreme learning machine (ELM) is an extremely fast learning method and has a powerful performance for pattern recognition tasks proven by enormous researches and engineers. However, its good generalization ability is built on large numbers of hidden neurons, which is not beneficial to real time response in the test process. In this paper, we proposed new ways, named "constrained extreme learning machines" (CELMs), to randomly select hidden neurons based on sample distribution. Compared to completely random selection of hidden nodes in ELM, the CELMs randomly select hidden nodes from the constrained vector space containing some basic combinations of original sample vectors. The experimental results show that the CELMs have better generalization ability than traditional ELM, SVM and some other related methods. Additionally, the CELMs have a similar fast learning speed as ELM.

Motivation & Objective

• To address the trade-off between generalization performance and computational efficiency in Extreme Learning Machines (ELM) for classification tasks.
• To reduce the number of hidden neurons required in ELM without sacrificing generalization ability.
• To improve upon the random selection of hidden neurons in ELM by introducing structural constraints based on sample distribution.
• To develop a method that maintains ELM’s fast learning speed while enhancing classification accuracy.
• To validate the proposed method against ELM, SVM, and other related algorithms on classification benchmarks.

Proposed method

• CELMs select hidden neurons from a constrained vector space constructed using basic combinations of original training samples.
• The constrained vector space is formed by linear combinations of input vectors, ensuring that selected neurons are representative of the underlying data distribution.
• Hidden neuron weights and biases are randomly generated but constrained to lie within this predefined space to improve representational quality.
• The output weights are computed using a least-squares solution, preserving the fast learning property of ELM.
• The method retains the single-layer feedforward network structure of ELM but enhances feature representation through structured initialization.
• The constrained selection process reduces redundancy and improves generalization by favoring neurons that capture salient data patterns.

Experimental results

Research questions

• RQ1 Can constraining the selection of hidden neurons in ELM improve generalization performance in classification tasks?
• RQ2 Does the proposed constrained selection method reduce the required number of hidden neurons while maintaining or improving accuracy?
• RQ3 How does the performance of CELMs compare to traditional ELM, SVM, and other related methods in terms of accuracy and speed?
• RQ4 To what extent does the structure of the constrained vector space influence the learning performance of CELMs?
• RQ5 Can CELMs maintain the fast learning speed of ELM while achieving better generalization?

Key findings

• CELMs achieve better generalization performance than traditional ELM across multiple classification datasets.
• The proposed method outperforms SVM and other related methods in terms of classification accuracy on benchmark datasets.
• CELMs maintain the fast learning speed characteristic of ELM, making them suitable for real-time applications.
• The constrained selection of hidden neurons leads to improved feature representation, reducing the need for large numbers of neurons.
• Experimental results show that CELMs achieve higher accuracy with fewer hidden neurons compared to standard ELM.
• The constrained vector space significantly enhances model performance by focusing on meaningful data combinations during neuron initialization.