[Paper Review] SynergicLearning: Neural Network-Based Feature Extraction for Highly-Accurate Hyperdimensional Learning

Mahdi Nazemi, Amirhossein Esmaili+2 more|arXiv (Cornell University)|Jul 30, 2020

Ferroelectric and Negative Capacitance Devices27 references4 citations

TL;DR

This paper proposes SynergicLearning, a hybrid machine learning model combining a neural network (NN) for feature extraction with a hyperdimensional (HD) computing classifier, enabling high accuracy (comparable to pure NNs), improved robustness over standalone HD models, and efficient on-chip deployment. The framework achieves 1.60x better power efficiency and 2.13x lower latency than state-of-the-art HD implementations, with a compiler and parameterized hardware mapping for real-time, incremental learning.

ABSTRACT

Machine learning models differ in terms of accuracy, computational/memory complexity, training time, and adaptability among other characteristics. For example, neural networks (NNs) are well-known for their high accuracy due to the quality of their automatic feature extraction while brain-inspired hyperdimensional (HD) learning models are famous for their quick training, computational efficiency, and adaptability. This work presents a hybrid, synergic machine learning model that excels at all the said characteristics and is suitable for incremental, on-line learning on a chip. The proposed model comprises an NN and a classifier. The NN acts as a feature extractor and is specifically trained to work well with the classifier that employs the HD computing framework. This work also presents a parameterized hardware implementation of the said feature extraction and classification components while introducing a compiler that maps any arbitrary NN and/or classifier to the aforementioned hardware. The proposed hybrid machine learning model has the same level of accuracy (i.e. $\pm$1%) as NNs while achieving at least 10% improvement in accuracy compared to HD learning models. Additionally, the end-to-end hardware realization of the hybrid model improves power efficiency by 1.60x compared to state-of-the-art, high-performance HD learning implementations while improving latency by 2.13x. These results have profound implications for the application of such synergic models in challenging cognitive tasks.

Motivation & Objective

• To bridge the gap between high-accuracy neural networks and the computational efficiency of hyperdimensional (HD) learning models.
• To enable end-to-end, on-chip deployment of a hybrid model suitable for incremental and online learning.
• To develop a compiler and parameterized hardware architecture that maps arbitrary neural networks and classifiers to the synergic framework.

Proposed method

• Train a neural network specifically to generate features optimized for subsequent classification using a hyperdimensional (HD) computing framework.
• Design a hybrid model where the NN acts as a feature extractor and the HD classifier performs final decision-making.
• Implement a parameterized hardware design for both feature extraction and HD classification components to support efficient on-chip execution.
• Develop a compiler that maps any arbitrary neural network and/or classifier to the proposed hardware-aware model.
• Optimize the system for low power and low latency by leveraging the inherent efficiency of HD computing and hardware-aware NN design.
• Integrate the entire pipeline into a single, compact, and scalable hardware architecture for real-time inference.

Experimental results

Research questions

• RQ1 Can a hybrid model combining neural networks and hyperdimensional computing achieve the accuracy of pure neural networks while matching the efficiency of HD models?
• RQ2 How does the synergic design improve accuracy compared to standalone HD learning models?
• RQ3 What level of power efficiency and latency reduction can be achieved through hardware-aware compilation and parameterized implementation?
• RQ4 Can the proposed compiler and hardware mapping support arbitrary neural networks and classifiers for flexible deployment?
• RQ5 How effective is the model for incremental, on-line learning in resource-constrained environments?

Key findings

• The hybrid model achieves the same accuracy as pure neural networks, within ±1%, demonstrating that HD-based classification does not compromise performance when paired with a well-trained feature extractor.
• The model improves accuracy by at least 10% compared to standalone hyperdimensional learning models, indicating a synergistic performance gain.
• The end-to-end hardware realization achieves 1.60x better power efficiency than state-of-the-art high-performance HD learning implementations.
• The hardware design reduces latency by 2.13x compared to the best existing HD learning systems.
• The proposed compiler and parameterized hardware enable efficient mapping of arbitrary neural networks and classifiers to the target architecture, supporting real-time and on-chip deployment.
• The framework is well-suited for incremental and online learning applications due to its low computational footprint and adaptability.