[Paper Review] User Reviews and Language: How Language Influences Ratings

Scott A. Hale|arXiv (Cornell University)|May 6, 2016

Digital Marketing and Social Media11 references3 citations

TL;DR

This study analyzes star ratings from multilingual user reviews of London tourist attractions on TripAdvisor, finding that ratings across languages are generally highly correlated but vary significantly by language pair. The key contribution is evidence that averaging ratings across all languages may be misleading, as some languages (e.g., German, French) correlate more strongly with others than do others (e.g., Japanese, Russian), suggesting language-specific relevance for decision-making.

ABSTRACT

The number of user reviews of tourist attractions, restaurants, mobile apps, etc. is increasing for all languages; yet, research is lacking on how reviews in multiple languages should be aggregated and displayed. Speakers of different languages may have consistently different experiences, e.g., different information available in different languages at tourist attractions or different user experiences with software due to internationalization/localization choices. This paper assesses the similarity in the ratings given by speakers of different languages to London tourist attractions on TripAdvisor. The correlations between different languages are generally high, but some language pairs are more correlated than others. The results question the common practice of computing average ratings from reviews in many languages.

Motivation & Objective

• To assess whether user reviews in different languages can be meaningfully aggregated into a single average rating for tourist attractions.
• To investigate the degree of correlation between star ratings given by speakers of different languages to the same London attractions.
• To evaluate the implications of language differences in user experiences and review behavior for multilingual e-commerce and interface design.
• To explore how cultural and linguistic factors may influence rating behavior and the perceived relevance of foreign-language reviews.

Proposed method

• Crawled 516,641 reviews of 3,040 London tourist attractions from tripadvisor.co.uk in July 2015.
• Identified the language of each review using the machine translation link parameter and validated with the Compact Language Detection (CLD) toolkit.
• Extracted star ratings, user IDs, review dates, and author locations for analysis.
• Computed pairwise correlation coefficients between ratings in different languages to assess similarity.
• Analyzed user behavior, including multilingual review writing and review frequency, to assess user activity patterns.
• Used statistical tests (p < 0.001) to compare review activity between monolingual and multilingual users.

Experimental results

Research questions

• RQ1 How similar are star ratings given by users speaking different languages to the same London tourist attractions?
• RQ2 Which language pairs show the highest and lowest correlations in user ratings?
• RQ3 To what extent do multilingual users differ in review behavior compared to monolingual users?
• RQ4 How does the correlation between language pairs affect the usefulness of foreign-language reviews for decision-making?

Key findings

• The correlation between ratings in different languages is generally high, with a median correlation of 0.68 across all language pairs.
• German, Norwegian, and French ratings show the strongest correlations with other languages, indicating higher cross-linguistic consistency.
• Japanese, Portuguese, and Russian ratings show the weakest correlations, suggesting lower predictive relevance for speakers of other languages.
• Users who write in multiple languages are more active, averaging 5.1 reviews per user compared to 3.8 for monolingual users (p < 0.001).
• 64% of users wrote only one review, and only 1% of users wrote in more than one language, though multilingual users were more active overall.
• The correlation structure implies that ratings from certain languages may be more indicative of a user’s potential experience than others, depending on their native language.