[Paper Review] Large Scale Analysis of Open MOOC Reviews to Support Learners' Course Selection

Manuel J. Gómez, Mario Calderón+3 more|arXiv (Cornell University)|Jan 11, 2022

Online Learning and Analytics4 citations

TL;DR

This study analyzes 2.4 million open MOOC reviews across five platforms using NLP techniques to improve course selection. It finds that numeric ratings are heavily biased (63% 5-star), sentiment analysis confirms positive sentiment in 80.8–83.8% of reviews, and topic modeling reveals key themes like course applicability and difficulty, enabling more transparent, data-driven course recommendations.

ABSTRACT

The recent pandemic has changed the way we see education. It is not surprising that children and college students are not the only ones using online education. Millions of adults have signed up for online classes and courses during last years, and MOOC providers, such as Coursera or edX, are reporting millions of new users signing up in their platforms. However, students do face some challenges when choosing courses. Though online review systems are standard among many verticals, no standardized or fully decentralized review systems exist in the MOOC ecosystem. In this vein, we believe that there is an opportunity to leverage available open MOOC reviews in order to build simpler and more transparent reviewing systems, allowing users to really identify the best courses out there. Specifically, in our research we analyze 2.4 million reviews (which is the largest MOOC reviews dataset used until now) from five different platforms in order to determine the following: (1) if the numeric ratings provide discriminant information to learners, (2) if NLP-driven sentiment analysis on textual reviews could provide valuable information to learners, (3) if we can leverage NLP-driven topic finding techniques to infer themes that could be important for learners, and (4) if we can use these models to effectively characterize MOOCs based on the open reviews. Results show that numeric ratings are clearly biased (63\% of them are 5-star ratings), and the topic modeling reveals some interesting topics related with course advertisements, the real applicability, or the difficulty of the different courses. We expect our study to shed some light on the area and promote a more transparent approach in online education reviews, which are becoming more and more popular as we enter the post-pandemic era.

Motivation & Objective

• Address the lack of standardized, transparent review systems in the MOOC ecosystem, which currently exclude 90% of non-completing learners.
• Investigate whether open reviews—especially textual ones—can provide discriminative, actionable insights for learners choosing courses.
• Develop NLP-driven models to extract sentiment and topic themes from open MOOC reviews for improved course characterization.
• Evaluate the alignment between sentiment, numeric ratings, and topic modeling to support evidence-based course recommendations.
• Promote a more transparent, decentralized, and learner-centered approach to MOOC review systems to counteract rating bias and choice overload.

Proposed method

• Collected and analyzed 2.4 million open MOOC reviews from five major platforms (e.g., Coursera, edX), including both numeric ratings and free-text reviews.
• Applied three pre-trained NLP sentiment analysis models (from Hugging Face, VADER, and TextBlob) to classify textual reviews into positive, neutral, or negative sentiment.
• Used Latent Dirichlet Allocation (LDA) for topic modeling on two distinct word sets: (1) qualitative descriptors (e.g., 'fun', 'informative'), and (2) content-related terms (e.g., 'programming', 'cloud computing').
• Calculated sentiment compound scores and correlated them with numeric ratings to assess consistency between quantitative and qualitative feedback.
• Mapped topic distributions across the dataset to identify frequently discussed themes and their sentiment associations.
• Validated model outputs by comparing topic clusters with course content and assessing sentiment-topic coherence in representative MOOCs.

Experimental results

Research questions

• RQ1 To what extent do numeric ratings in MOOCs provide discriminative information for learners, given potential rating inflation?
• RQ2 Can NLP-driven sentiment analysis on textual reviews offer reliable and complementary insights to numeric ratings?
• RQ3 Can topic modeling on open reviews uncover meaningful, learner-driven themes related to course quality, applicability, or difficulty?
• RQ4 How do sentiment and topic patterns differ between high- and low-rated MOOCs, and can they be used to characterize courses more effectively?
• RQ5 Can the integration of sentiment and topic modeling results improve course recommendation systems and reduce learner search costs?

Key findings

• 63% of numeric ratings were 5-star, indicating a significant positive bias in the rating system that distorts course perception.
• 80.8% to 83.8% of textual reviews were classified as positive by three different sentiment analysis models, confirming a strong alignment between sentiment and high ratings.
• Topic modeling on qualitative descriptors revealed recurring themes such as 'informative_easy_fun', 'money_worth_ad', and 'slide_powerpoint_visual', reflecting learner perceptions of course experience.
• Content-based topic modeling identified high-frequency themes like 'Health and lifestyle', 'Programming', and 'Cloud computing', offering insight into course focus areas.
• Courses with higher sentiment scores were significantly more likely to be associated with positive topics (e.g., 'real_worth_life'), while negative sentiment correlated with topics like 'error_week_wrong'.
• The integration of sentiment and topic modeling enabled effective characterization of specific MOOCs, demonstrating that both metrics can be used together to support transparent, data-driven course selection.