[Paper Review] Multi-modal Sentiment Analysis using Super Characters Method on Low-power CNN Accelerator Device

Baohua Sun, Lin Yang+2 more|arXiv (Cornell University)|Jan 28, 2020

Sentiment Analysis and Opinion Mining8 references4 citations

TL;DR

This paper proposes a modified Super Characters method for multi-modal sentiment analysis on low-power CNN accelerator hardware, integrating text and tabular data into a 2D image representation for efficient inference. It achieves up to 80.82% accuracy on the General_support task using a 300mW CNN chip, demonstrating feasibility for real-world, power-efficient NLP deployment.

ABSTRACT

Recent years NLP research has witnessed the record-breaking accuracy improvement by DNN models. However, power consumption is one of the practical concerns for deploying NLP systems. Most of the current state-of-the-art algorithms are implemented on GPUs, which is not power-efficient and the deployment cost is also very high. On the other hand, CNN Domain Specific Accelerator (CNN-DSA) has been in mass production providing low-power and low cost computation power. In this paper, we will implement the Super Characters method on the CNN-DSA. In addition, we modify the Super Characters method to utilize the multi-modal data, i.e. text plus tabular data in the CL-Aff sharedtask.

Motivation & Objective

• Address the challenge of deploying high-accuracy NLP models in power-constrained environments.
• Extend the Super Characters method to handle multi-modal inputs (text and tabular data) for sentiment classification.
• Demonstrate the feasibility of running DNN-based sentiment analysis on low-power, low-cost CNN accelerator chips.
• Improve model generalization on small, skewed datasets through fine-tuning and data augmentation strategies.
• Evaluate performance on the CL-AFF shared task dataset, which includes six binary classification tasks on Reddit comments with metadata.

Proposed method

• Convert text input into a 2D Super Characters image using the Squared English Word (SEW) method for Latin scripts.
• Integrate tabular data (e.g., wordcount, created_utc) into a separate region of the 2D image for multi-modal fusion.
• Train a CNN model using a Model Development Kit (MDK) on the CNN Domain-Specific Accelerator (CNN-DSA), then deploy the fixed-point model via the Software Development Kit (SDK).
• Use 10-fold cross-validation to train and evaluate six separate binary classification models for the six sentiment tasks.
• Optimize image layout based on sentence length distribution to maximize feature representation within 224×224 resolution constraints.
• Leverage transfer learning by fine-tuning pre-trained models on the larger dataset to improve performance on the smaller CL-AFF dataset.

Experimental results

Research questions

• RQ1 Can the Super Characters method be effectively extended to handle multi-modal inputs combining text and tabular data?
• RQ2 How does the performance of the Super Characters method compare on low-power CNN accelerator hardware for multi-modal sentiment analysis tasks?
• RQ3 What impact does model fine-tuning and data augmentation have on performance in low-resource, skewed datasets like CL-AFF?
• RQ4 Can the Super Characters approach achieve competitive accuracy while maintaining low power consumption (300mW) and high inference speed (140fps) on edge devices?
• RQ5 How does the integration of author identity and metadata improve sentiment classification beyond raw text alone?

Key findings

• The best-performing model, Design Option Four, achieved 80.82% accuracy on the General_support task, outperforming baseline predictions.
• Design Option Four showed slightly better performance than Design Option One across all tasks, with the highest F1 score of 59.29% on the Emotion_support task.
• The model achieved 140fps inference speed with only 300mW power consumption on a low-power CNN accelerator chip, confirming energy efficiency.
• Despite improvements, overall accuracy remained below 80% across most tasks, indicating challenges with small, imbalanced datasets.
• The use of fine-tuned pre-trained models and data augmentation (e.g., synonym replacement) is expected to further improve performance on such low-resource datasets.
• The results were less accurate than those reported on larger datasets like Wikipedia, highlighting the need for better data scaling or augmentation techniques.