[Paper Review] Analysis of player's in-game performance vs rating: Case study of Heroes of Newerth

Neven Čaplar, Mirko Sužnjević+1 more|arXiv (Cornell University)|May 22, 2013

Peer-to-Peer Network Technologies10 references3 citations

TL;DR

This study evaluates the Heroes of Newerth (HoN) matchmaking rating (MMR) system by correlating in-game performance metrics—such as APM, kills, deaths, assists, denies, and account age—with assigned player ratings. Using a dataset of 338,681 players, the authors find that while MMR generally reflects player skill, exploitable weaknesses exist, including intentional MMR deflation and slow skill placement, enabling 'smurfs' and low-skill players to rank higher than deserved.

ABSTRACT

We evaluate the rating system of "Heroes of Newerth" (HoN), a multiplayer online action role-playing game, by using statistical analysis and comparison of a player's in-game performance metrics and the player rating assigned by the rating system. The datasets for the analysis have been extracted from the web sites that record the players' ratings and a number of empirical metrics. Results suggest that the HoN's Matchmaking rating algorithm, while generally capturing the skill level of the player well, also has weaknesses, which have been exploited by players to achieve a higher placement on the ranking ladder than deserved by actual skill. In addition, we also illustrate the effects of the choice of the business model (from pay-to-play to free-to-play) on player population.

Motivation & Objective

• To evaluate the accuracy of HoN’s matchmaking rating (MMR) system in reflecting players’ true skill levels.
• To identify exploitable behaviors that allow players to achieve higher rankings than their actual in-game performance warrants.
• To examine the impact of game business models (pay-to-play vs. free-to-play) on player population distribution and rating progression.
• To assess the speed and consistency of MMR placement for new players based on performance metrics and account age.
• To improve understanding of player behavior patterns and their effects on rating system integrity.

Proposed method

• Collected performance data from public HoN player databases, including MMR, APM, kill/death ratio, assists, denies, and account age.
• Used statistical correlation analysis to compare in-game metrics (e.g., APM, kills, denies) with MMR across a large player sample.
• Applied heat maps and scatter plots to visualize relationships between MMR and variables like account age and game count.
• Conducted sample consistency checks by generating multiple random subsets of 3,000+ players and re-evaluating APM-MM R correlations.
• Identified behavioral anomalies such as win/loss and kill/death ratio inconsistencies to detect potential MMR manipulation.
• Analyzed historical data to correlate shifts in business model (pay-to-play to free-to-play) with changes in player population and rating distribution.

Experimental results

Research questions

• RQ1 To what extent does the HoN MMR system accurately reflect a player’s true in-game performance?
• RQ2 Which in-game metrics (e.g., APM, denies, kills) show the strongest correlation with MMR, and how do they vary across skill tiers?
• RQ3 Are there exploitable behaviors—such as intentional MMR deflation or win trading—that allow players to bypass skill-based placement?
• RQ4 How did the transition from a pay-to-play to a free-to-play model affect player distribution and rating progression over time?
• RQ5 How quickly does the MMR system place new players into their correct skill brackets, and what factors influence this process?

Key findings

• The MMR system generally correlates well with in-game performance, particularly with metrics like APM, kills, and denies, which show strong positive trends with increasing MMR.
• Highly ranked players consistently perform better in denies, with nearly all top-tier players denying creeps, indicating that this mechanic is a key differentiator of skilled play.
• A significant number of players exhibit anomalous behavior, such as low kill/death ratios paired with high win/loss ratios, suggesting intentional MMR manipulation or team-based exploits.
• The transition from pay-to-play to free-to-play in July 2011 led to a sharp increase in new accounts (around 400 days prior to data collection), visible as a surge in the account age distribution.
• Players with very recent accounts (<50 days) are rarely found in high MMR brackets, indicating that rapid MMR inflation is uncommon and the system is relatively slow to place players in correct skill tiers.
• Sample consistency tests show minimal variation across multiple random subsets, confirming that the observed correlations are stable and not artifacts of sampling bias.