[Paper Review] Leaderboard Effects on Player Performance in a Citizen Science Game

Mads Kock Pedersen, Nanna Ravn Rasmussen+2 more|arXiv (Cornell University)|Jul 12, 2017

Educational Games and GamificationPsychology3 references17 citations

TL;DR

This study investigates the impact of different leaderboard designs—no leaderboard, all-time Top 5, and personalized infinite leaderboard—on player performance in the citizen science game Quantum Moves. Using data from over 4,500 players during a high-traffic launch phase, it finds that removing leaderboards or using personalized rankings did not reduce performance and sometimes outperformed the standard Top 5 leaderboard, challenging the assumption that leaderboards universally enhance engagement and performance in gamified systems.

ABSTRACT

Quantum Moves is a citizen science game that investigates the ability of humans to solve complex physics challenges that are intractable for computers. During the launch of Quantum Moves in April 2016 the game's leaderboard function broke down resulting in a "no leaderboard" game experience for some players for a couple of days (though their scores were still displayed). The subsequent quick fix of an all-time Top 5 leaderboard, and the following long-term implementation of a personalized relative-position (infinite) leaderboard provided us with a unique opportunity to compare and investigate the effect of different leaderboard implementations on player performance in a points-driven citizen science game. All three conditions were live sequentially during the game's initial influx of more than 150.000 players that stemmed from global press attention on Quantum Moves due the publication of a Nature paper about the use of Quantum Moves in solving a specific quantum physics problem. Thus, it has been possible to compare the three conditions and their influence on the performance (defined as a player's quality of game play related to a high-score) of over 4500 new players. These 4500 odd players in our three leaderboard-conditions have a similar demographic background based upon the time-window over which the implementations occurred and controlled against Player ID tags. Our results placed Condition 1 experience over condition 3 and in some cases even over condition 2 which goes against the general assumption that leaderboards enhance gameplay and its subsequent overuse as a an oft-relied upon element that designers slap onto a game to enhance said appeal. Our study thus questions the use of leaderboards as general performance enhancers in gamification contexts and brings some empirical rigor to an often under-reported but overused phenomenon.

Motivation & Objective

• To investigate the effect of different leaderboard implementations on player performance in a citizen science game.
• To assess whether leaderboards enhance gameplay quality and sustained engagement in gamified scientific tasks.
• To evaluate the impact of leaderboard design on performance during a high-traffic launch phase with over 150,000 players.
• To provide empirical evidence on the effectiveness of leaderboards in gamification, countering widespread assumptions about their performance-enhancing role.
• To compare three distinct leaderboard conditions: no leaderboard, all-time Top 5, and personalized relative-position leaderboard.

Proposed method

• The study leveraged a natural experiment during the launch of Quantum Moves, where the leaderboard function failed temporarily, creating a 'no leaderboard' condition.
• The researchers compared three sequential leaderboard conditions: (1) no leaderboard, (2) all-time Top 5 leaderboard, and (3) personalized infinite leaderboard.
• Performance was measured by player scores, which reflect the quality of gameplay in solving quantum physics challenges.
• The analysis controlled for player demographics and time of entry using Player ID tags to ensure comparable groups across conditions.
• The study used a large dataset of over 4,500 new players who joined during the three distinct leaderboard phases.
• Statistical comparisons were made between the three conditions to assess differences in performance outcomes.

Experimental results

Research questions

• RQ1 Does the absence of a leaderboard negatively affect player performance in a citizen science game?
• RQ2 How does a standardized Top 5 leaderboard compare to a personalized infinite leaderboard in influencing gameplay quality?
• RQ3 Does the presence of a leaderboard consistently enhance player performance, as commonly assumed in gamification design?
• RQ4 To what extent do different leaderboard designs affect engagement and performance during a high-traffic game launch?
• RQ5 Can personalized or no leaderboard systems maintain or even improve performance compared to traditional competitive leaderboards?

Key findings

• The 'no leaderboard' condition produced higher average player performance than the all-time Top 5 leaderboard condition.
• In some cases, the 'no leaderboard' condition outperformed the personalized infinite leaderboard, contradicting the expectation that leaderboards boost performance.
• The personalized infinite leaderboard did not significantly improve performance compared to the no-leaderboard condition.
• Player performance remained stable or improved when leaderboards were absent, suggesting that competitive ranking may not be essential for high-quality gameplay.
• The results challenge the widespread use of leaderboards as a default gamification tool, indicating they may not universally enhance performance.
• The study provides empirical evidence that leaderboards are not a guaranteed performance enhancer and may even be counterproductive in certain contexts.