[Paper Review] Comprehensive Analysis of Aspect Term Extraction Methods using Various Text Embeddings

Łukasz Augustyniak, Tomasz Kajdanowicz+1 more|arXiv (Cornell University)|Sep 11, 2019

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper presents a comprehensive empirical study comparing LSTM and BiLSTM architectures with various pre-trained word embeddings and character-level embeddings for aspect term extraction (ATE) on SemEval datasets. It demonstrates that BiLSTM with word and character embeddings achieves state-of-the-art performance comparable to complex models, offering a robust, efficient baseline for ATE with minimal architectural complexity.

ABSTRACT

Recently, a variety of model designs and methods have blossomed in the context of the sentiment analysis domain. However, there is still a lack of wide and comprehensive studies of aspect-based sentiment analysis (ABSA). We want to fill this gap and propose a comparison with ablation analysis of aspect term extraction using various text embedding methods. We particularly focused on architectures based on long short-term memory (LSTM) with optional conditional random field (CRF) enhancement using different pre-trained word embeddings. Moreover, we analyzed the influence on the performance of extending the word vectorization step with character embedding. The experimental results on SemEval datasets revealed that not only does bi-directional long short-term memory (BiLSTM) outperform regular LSTM, but also word embedding coverage and its source highly affect aspect detection performance. An additional CRF layer consistently improves the results as well.

Motivation & Objective

• To address the lack of comprehensive, reproducible comparisons of ATE methods using diverse embeddings and architectures.
• To evaluate the robustness and coverage of pre-trained word embeddings in domain-specific ATE tasks.
• To investigate the impact of character-level embeddings in improving performance for low-resource or underrepresented languages.
• To provide a practical, data-driven guideline for selecting optimal ATE model configurations based on performance, complexity, and resource constraints.
• To establish a strong, simple baseline using BiLSTM with hybrid word and character embeddings that rivals more complex models.

Proposed method

• Employs bidirectional and unidirectional LSTM networks as the core sequence modeling architecture for ATE.
• Integrates multiple pre-trained word embeddings (GloVe, fastText, word2vec, NumberBatch, Amazon Reviews) from diverse corpora and sizes.
• Enhances word representations with character-level embeddings using a separate CNN-based encoder for subword information.
• Applies CRF layer on top of LSTM to model label dependencies and improve sequence labeling accuracy.
• Trains and evaluates models on two SemEval benchmark datasets: Restaurant and Laptops.
• Conducts ablation studies across embedding types, network architectures, and CRF integration to isolate performance factors.

Experimental results

Research questions

• RQ1 How robust are general-domain pre-trained word embeddings in domain-specific aspect term extraction?
• RQ2 How does the coverage of word embeddings impact ATE performance across different datasets and languages?
• RQ3 In what scenarios do character-based word embeddings improve performance over static word embeddings alone?
• RQ4 Which combination of architecture (LSTM/BiLSTM), embeddings (word, character), and CRF integration yields optimal ATE performance?
• RQ5 Can a simple BiLSTM with hybrid word and character embeddings outperform more complex models?

Key findings

• BiLSTM with word and character embeddings (WoCh-BiLSTM-CRF) achieved the highest F1 score of 85.2% on the Restaurant dataset, outperforming more complex models.
• The WoCh-BiLSTM-CRF model achieved 80.13% F1 on the Laptops dataset, significantly outperforming other configurations and showing strong generalization.
• GloVe.840B and fastText-crawl embeddings consistently ranked among the top-performing word embeddings across both datasets.
• Incorporating character-level embeddings improved performance by up to 2.5% F1 over word-only models, especially benefiting low-resource or rare words.
• The addition of CRF layer consistently improved F1 scores across all configurations, with the largest gains observed in BiLSTM models.
• Simple BiLSTM-based models with hybrid embeddings matched or exceeded the performance of more complex models using large-scale language models, proving their efficiency and effectiveness.