[Paper Review] Replication issues in syntax-based aspect extraction for opinion mining

Edison Marrese-Taylor, Yutaka Matsuo|arXiv (Cornell University)|Jan 6, 2017

Topic Modeling3 citations

TL;DR

This paper investigates replication challenges in syntax-based aspect extraction for opinion mining, demonstrating that insufficient documentation of preprocessing and hyperparameters prevents accurate reproduction of published results. Despite using standard tools and datasets, the authors fail to replicate state-of-the-art performance, highlighting the critical need for code sharing to ensure methodological transparency and scientific validity in NLP research.

ABSTRACT

Reproducing experiments is an important instrument to validate previous work and build upon existing approaches. It has been tackled numerous times in different areas of science. In this paper, we introduce an empirical replicability study of three well-known algorithms for syntactic centric aspect-based opinion mining. We show that reproducing results continues to be a difficult endeavor, mainly due to the lack of details regarding preprocessing and parameter setting, as well as due to the absence of available implementations that clarify these details. We consider these are important threats to validity of the research on the field, specifically when compared to other problems in NLP where public datasets and code availability are critical validity components. We conclude by encouraging code-based research, which we think has a key role in helping researchers to understand the meaning of the state-of-the-art better and to generate continuous advances.

Motivation & Objective

• To investigate the replicability of three well-known syntax-based aspect extraction methods in opinion mining.
• To assess whether insufficient documentation of preprocessing and hyperparameters hinders reproduction of published results.
• To evaluate the impact of parameter tuning and pre-processing choices on model performance.
• To advocate for code-based research practices to improve transparency and reproducibility in NLP.

Proposed method

• The authors reimplemented three prominent syntax-based aspect extraction models: FBA, TBA, and a third method based on dependency parsing and rule-based filtering.
• They applied standard NLP pipelines using Senna and Stanford CoreNLP for tokenization, POS tagging, and dependency parsing.
• Preprocessing steps such as stemming, stopword removal, and normalization were systematically varied to assess their impact on performance.
• Hyperparameters such as minimum support ($min_{sup}$) and confidence thresholds ($t$) were tuned across multiple configurations to identify optimal settings.
• Performance was evaluated using standard metrics: precision, recall, and F1-score on benchmark datasets including the Customer Review Dataset.
• The authors compared their results against original reported scores, analyzing discrepancies through controlled ablation studies.

Experimental results

Research questions

• RQ1 Are the explanations provided in original papers sufficient to reproduce the reported results in syntax-based aspect extraction?
• RQ2 To what extent do differences in preprocessing pipelines affect the performance of aspect extraction models?
• RQ3 How critical is hyperparameter tuning in achieving state-of-the-art performance, and are the reported parameters optimal?
• RQ4 Why do the authors fail to reproduce the original results despite using the same datasets and standard NLP tools?

Key findings

• The authors were unable to reproduce the reported F1 scores for any of the three models, with performance consistently falling short of original claims.
• The F1 score for the FBA model dropped to 0.368 in the authors' implementation, compared to the original 0.426, indicating a significant performance gap.
• Parameter tuning revealed that $min_{sup}$ in FBA and the confidence threshold $t$ in TBA had a major impact on performance, suggesting that original settings were not optimal.
• Preprocessing choices such as stemming and stopword removal had inconsistent effects across models, with no single pipeline yielding best results universally.
• The lack of clarity on whether gold-standard annotations were stemmed led to ambiguity in evaluation, raising concerns about the validity of original comparisons.
• The absence of source code and detailed implementation details made reverse-engineering the original methods extremely difficult, even with access to datasets.