[Paper Review] Towards a Data-Driven Requirements Engineering Approach: Automatic Analysis of User Reviews

Jialiang Wei, Anne-Lise Courbis+4 more|arXiv (Cornell University)|Jun 30, 2022

Web Data Mining and Analysis16 references4 citations

TL;DR

This paper proposes a data-driven requirements engineering approach that automatically classifies French user reviews from health and fitness apps into four labels—rating, bug report, feature request, and user experience—using fine-tuned CamemBERT, a state-of-the-art French BERT model. The method achieves promising performance on a newly created multi-label dataset of 6,000 manually annotated reviews, demonstrating the feasibility of automated feature request detection from user feedback.

ABSTRACT

6000 French user reviews from three applications on Google Play (Garmin Connect, Huawei Health, Samsung Health) are labelled manually. We selected four labels: rating, bug report, feature request and user experience. Ratings are simple text which express the overall evaluation to that app, including praise, criticism, or dissuasion. Bug reports show the problems that users have met while using the app, like loss of data, crash of app, connection error, etc. Feature requests reflect the demande of users on new function, new content, new interface, etc. In user experience, users describe their experience in relation to the functionality of the app, how does certain functions be helpful. As we can observe from the following table, that shows examples of labelled user reviews, each review belongs to one or more categories. App Total Rating Bug report Feature request User experience Garmin Connect 2000 1260 757 170 493 Huawei Health 2000 1068 819 384 289 Samsung Health 2000 1324 491 486 349 New Dataset Based on this dataset, we developed a labeled dataset containing 6,000 English and 6,000 French reviews for classification, as well as 1,200 bilingual reviews for clustering. The new dataset has been made publicly available on Zenodo at the following link: https://zenodo.org/records/11066414

Motivation & Objective

• To address the lack of publicly available, high-quality French user review datasets for NLP tasks in requirements engineering.
• To develop an automated, data-driven approach to extract actionable requirements from user reviews on app stores.
• To evaluate the effectiveness of CamemBERT for multi-label classification of French app reviews in the Health & Fitness domain.
• To support software development teams in prioritizing feature requests and improving requirements elicitation through scalable, automated analysis.

Proposed method

• The authors collected 6,000 French user reviews from three health and fitness apps—Garmin Connect, Huawei Health, and Samsung Health—on Google Play.
• Each review was manually annotated with up to four labels: rating, bug report, feature request, and user experience, forming a multi-label classification dataset.
• A CamemBERT-based deep learning model was fine-tuned on the annotated dataset for multi-label text classification.
• The model leverages the bidirectional Transformer architecture to capture contextual semantics in French text, enabling accurate classification of complex user feedback.
• The dataset was shared publicly to support future research in French NLP and data-driven requirements engineering.

Experimental results

Research questions

• RQ1 Can CamemBERT effectively classify French app reviews into multiple labels relevant to requirements engineering?
• RQ2 How well does a fine-tuned CamemBERT model perform on a newly created French user review dataset for multi-label classification?
• RQ3 To what extent can automated analysis of user reviews support feature request identification in health and fitness applications?
• RQ4 Is there a significant gap in existing datasets for French app reviews, and can a new dataset improve the performance of NLP models in this domain?

Key findings

• The authors successfully created and released a multi-label French user review dataset of 6,000 reviews from three health and fitness apps, filling a critical gap in the literature.
• The CamemBERT-based model achieved strong performance in classifying user reviews into four categories: rating, bug report, feature request, and user experience.
• The dataset shows that 757 out of 2,000 reviews from Garmin Connect were labeled as feature requests, indicating a high volume of user-driven functionality demands.
• The study demonstrates that deep learning models like CamemBERT can effectively extract requirements from unstructured user feedback, especially for feature requests.
• The multi-label nature of the dataset allows for overlapping labels, reflecting real-world user feedback complexity, and supports nuanced analysis of user needs.
• The results suggest that automated analysis of user reviews can significantly reduce manual effort in requirements elicitation while improving software evolution processes.