[Paper Review] Challenges for Open-domain Targeted Sentiment Analysis

Luo Yun, Hongjie Cai+4 more|arXiv (Cornell University)|Apr 14, 2022

Topic Modeling5 citations

TL;DR

This paper introduces a new open-domain targeted sentiment analysis dataset with 6,013 human-labeled documents across six diverse domains, extending beyond sentence-level analysis to document-level sentiment extraction using a nested target annotation schema. It employs a sequence-to-sequence BART-based model for end-to-end generation, revealing significant challenges in handling long documents, complex target structures, and domain variations, with benchmark results showing substantial room for improvement despite state-of-the-art pre-trained models.

ABSTRACT

Since previous studies on open-domain targeted sentiment analysis are limited in dataset domain variety and sentence level, we propose a novel dataset consisting of 6,013 human-labeled data to extend the data domains in topics of interest and document level. Furthermore, we offer a nested target annotation schema to extract the complete sentiment information in documents, boosting the practicality and effectiveness of open-domain targeted sentiment analysis. Moreover, we leverage the pre-trained model BART in a sequence-to-sequence generation method for the task. Benchmark results show that there exists large room for improvement of open-domain targeted sentiment analysis. Meanwhile, experiments have shown that challenges remain in the effective use of open-domain data, long documents, the complexity of target structure, and domain variances.

Motivation & Objective

• To address the limitations of existing open-domain targeted sentiment analysis datasets, which are restricted to sentence-level analysis and narrow domains such as restaurant, laptop, and Twitter reviews.
• To extend the scope of targeted sentiment analysis to document-level input, capturing co-reference and discourse-level relationships that are lost in sentence-level processing.
• To improve practicality by introducing a nested target annotation schema that captures complex, hierarchical sentiment structures, such as relations between targets like 'Italian restaurant' and 'price'.
• To evaluate the effectiveness of pre-trained sequence-to-sequence models like BART in end-to-end open-domain targeted sentiment analysis under realistic, diverse, and long-form text conditions.
• To identify persistent challenges in open-domain sentiment analysis, including domain variance, long document complexity, and structural target ambiguity.

Proposed method

• Proposes a new dataset with 6,013 human-annotated documents spanning six domains: books, clothing, restaurants, hotels, news, and PhraseBank, each with diverse linguistic genres and document lengths.
• Introduces a nested target annotation schema that captures not only sentiment-bearing spans but also hierarchical and relational structures among targets, such as 'Italian restaurant' influencing 'price' and 'food'.
• Formulates open-domain targeted sentiment analysis as a sequence-to-sequence generation task using the pre-trained BART model, where the input is a document and the output is a structured list of (target, sentiment) pairs.
• Employs both pipeline and joint modeling approaches: pipeline separates target extraction and sentiment classification, while joint models perform both tasks end-to-end in a single generation step.
• Uses micro-F1 and Kappa scores to evaluate inter-annotator agreement, ensuring high-quality labeling across domains with varying target densities.
• Conducts ablation studies and case analyses to compare model performance across domains with different average target counts and structural complexities.

Experimental results

Research questions

• RQ1 How does the performance of open-domain targeted sentiment analysis models vary across diverse, real-world document domains beyond traditional sentence-level datasets?
• RQ2 To what extent does the nested target annotation schema improve the capture of complex, hierarchical sentiment relationships compared to traditional span-based labeling?
• RQ3 How do pipeline and joint modeling approaches compare in terms of performance and robustness when handling long documents and high target density?
• RQ4 What are the key challenges in applying pre-trained sequence-to-sequence models to open-domain targeted sentiment analysis, particularly regarding co-reference resolution and sentiment aggregation?
• RQ5 How do domain-specific variations and document length affect the generalization and reliability of sentiment analysis models in open-domain settings?

Key findings

• The proposed dataset spans six diverse domains with varying target densities, from 1.23 targets per document in PhraseBank to 5.03 in restaurants, reflecting real-world document complexity.
• Joint models outperform pipeline models in domains with low average target counts (e.g., Books, 2.50 targets per document), but pipeline models perform better in high-target domains (e.g., Restaurant, 5.03 targets per document).
• Benchmark results show significant performance gaps, indicating that even with state-of-the-art pre-trained models, open-domain targeted sentiment analysis remains a challenging task with substantial room for improvement.
• Case studies reveal that models struggle with anaphora resolution (e.g., failing to link 'they' or 'it' to their referents) and miss complex relational sentiment structures when target density is high.
• The study confirms that challenges remain in effectively utilizing open-domain data, handling long documents, managing complex target structures, and generalizing across domain variances.
• Inter-annotator agreement is high (F1 > 70% in most domains), validating the quality and consistency of the human-annotated dataset.