[Paper Review] Mining fine-grained opinions on closed captions of YouTube videos with an attention-RNN

Edison Marrese-Taylor, Jorge Balazs+1 more|arXiv (Cornell University)|Aug 8, 2017

Sentiment Analysis and Opinion Mining17 references3 citations

TL;DR

This paper introduces the first dataset, Youtubean, of YouTube video review closed captions for fine-grained opinion mining, and proposes an attention-RNN model for joint aspect extraction and sentiment classification. The model achieves state-of-the-art performance on SemEval ABSA and outperforms baselines on Youtubean, though all models show significant performance drops on the video review data, indicating domain differences between spoken and written text in opinion mining.

ABSTRACT

Video reviews are the natural evolution of written product reviews. In this paper we target this phenomenon and introduce the first dataset created from closed captions of YouTube product review videos as well as a new attention-RNN model for aspect extraction and joint aspect extraction and sentiment classification. Our model provides state-of-the-art performance on aspect extraction without requiring the usage of hand-crafted features on the SemEval ABSA corpus, while it outperforms the baseline on the joint task. In our dataset, the attention-RNN model outperforms the baseline for both tasks, but we observe important performance drops for all models in comparison to SemEval. These results, as well as further experiments on domain adaptation for aspect extraction, suggest that differences between speech and written text, which have been discussed extensively in the literature, also extend to the domain of product reviews, where they are relevant for fine-grained opinion mining.

Motivation & Objective

• To address the lack of research on fine-grained opinion mining in video product reviews, particularly from spoken language in YouTube unboxing and review videos.
• To create the first annotated dataset, Youtubean, from closed captions of YouTube product review videos to support research in spoken-language opinion mining.
• To develop an attention-RNN model that jointly performs aspect extraction and sentiment classification, leveraging attention to model compositional sentiment.
• To investigate the impact of domain differences between spoken (video) and written (textual) reviews on opinion mining performance.
• To evaluate whether domain adaptation techniques can mitigate performance drops when transferring models trained on written text to spoken video reviews.

Proposed method

• Annotate closed captions from YouTube product review videos to create the Youtubean dataset, focusing on aspect-sentiment pairs.
• Design an attention-augmented RNN model that jointly performs aspect extraction and sentiment classification using a shared encoder with attention over word representations.
• Use a soft attention mechanism to compute a convex combination of word representations, enabling the model to focus on relevant words for sentiment composition.
• Train the model end-to-end on the SemEval ABSA dataset and Youtubean, using cross-entropy loss for both aspect labeling and sentiment classification.
• Evaluate the model using F1-score on aspect extraction (AE), sentiment classification (SC), and joint aspect-sentiment classification (AESC), comparing against RNN and BiLSTM baselines.
• Apply domain adaptation techniques, such as fine-tuning on Youtubean and using pre-trained embeddings from different sources, to assess transferability between written and spoken review domains.

Experimental results

Research questions

• RQ1 How does the performance of opinion mining models trained on written product reviews generalize to spoken-language video reviews, as captured in YouTube closed captions?
• RQ2 Can an attention-RNN model achieve state-of-the-art performance on joint aspect extraction and sentiment classification without relying on hand-crafted linguistic features?
• RQ3 To what extent do domain adaptation techniques improve model performance when transferring from written to spoken review domains?
• RQ4 What are the key differences in linguistic patterns between written and spoken product reviews that affect opinion mining performance?
• RQ5 Why do models exhibit particularly poor performance on the negative sentiment class in video review data, and can attention mechanisms help mitigate this?

Key findings

• The attention-RNN model achieves state-of-the-art performance on the SemEval ABSA benchmark for joint aspect extraction and sentiment classification, outperforming previous models without requiring hand-crafted features.
• On the Youtubean dataset, the attention-RNN outperforms the baseline RNN for both aspect extraction and joint classification, with F1 scores of 52.32 and 48.05 respectively when using Wikipedia embeddings.
• All models, including the best-performing attention-RNN, show significant performance drops on Youtubean compared to SemEval, indicating a domain shift between written and spoken reviews.
• Domain adaptation techniques applied to the baseline RNNs improved performance by up to 3.56 F1-score on aspect extraction, but gains were limited and inconsistent across models.
• The model struggles most with the negative sentiment class, suggesting challenges related to data imbalance or compositional negation in spoken language.
• Senna embeddings did not consistently outperform other embeddings on Youtubean, indicating that general-purpose embeddings may be more suitable for spoken-language opinion mining than task-specific ones.