[Paper Review] A Comparison of Methods for Player Clustering via Behavioral Telemetry

Anders Drachen, Christian Thurau+2 more|arXiv (Cornell University)|Jul 15, 2014

Sports Analytics and Performance28 references50 citations

TL;DR

This paper evaluates unsupervised clustering methods—k-means, c-means, NMF, PCA, and Archetypal Analysis (AA)—on playtime and leveling speed telemetry from 70,014 World of Warcraft players over five years. AA outperforms others by generating interpretable, behaviorally valid archetypical profiles that correspond to real in-game behaviors, such as rapid level progression tied to expansion releases, while PCA and NMF produce counterintuitive results like players losing levels.

ABSTRACT

The analysis of user behavior in digital games has been aided by the introduction of user telemetry in game development, which provides unprecedented access to quantitative data on user behavior from the installed game clients of the entire population of players. Player behavior telemetry datasets can be exceptionally complex, with features recorded for a varying population of users over a temporal segment that can reach years in duration. Categorization of behaviors, whether through descriptive methods (e.g. segmention) or unsupervised/supervised learning techniques, is valuable for finding patterns in the behavioral data, and developing profiles that are actionable to game developers. There are numerous methods for unsupervised clustering of user behavior, e.g. k-means/c-means, Non-negative Matrix Factorization, or Principal Component Analysis. Although all yield behavior categorizations, interpretation of the resulting categories in terms of actual play behavior can be difficult if not impossible. In this paper, a range of unsupervised techniques are applied together with Archetypal Analysis to develop behavioral clusters from playtime data of 70,014 World of Warcraft players, covering a five year interval. The techniques are evaluated with respect to their ability to develop actionable behavioral profiles from the dataset.

Motivation & Objective

• To evaluate the effectiveness of unsupervised clustering techniques in generating actionable behavioral profiles from large-scale game telemetry.
• To address the challenge of interpretability in player clustering, especially when methods produce counterintuitive or infeasible behavioral profiles.
• To assess whether Archetypal Analysis (AA) via Simplex Volume Maximization (SIVM) produces more interpretable and behaviorally valid clusters than traditional methods like k-means, NMF, and PCA.
• To compare the distribution of players across clusters and evaluate the suitability of hard vs. soft clustering assignments for game design and monetization strategies.

Proposed method

• Applies k-means, c-means, NMF, PCA, and Archetypal Analysis (AA) to playtime and leveling speed data from 70,014 World of Warcraft players over five years.
• Uses Simplex Volume Maximization (SIVM) to compute AA basis vectors that represent extreme, archetypical player behaviors on the convex hull of the data distribution.
• Evaluates interpretability by checking whether basis vectors correspond to actual, legal in-game behaviors (e.g., no level loss, instantaneous level-ups at expansions).
• Performs hard clustering assignments by assigning each player to the nearest basis vector or centroid to enable quantitative profiling.
• Compares the resulting cluster distributions across methods using visualizations (e.g., Fig. 4) and assesses structural differences in cluster composition.
• Validates results by correlating archetypical profiles with known game events, such as expansion pack releases, to confirm behavioral plausibility.

Experimental results

Research questions

• RQ1 Which clustering method produces the most interpretable behavioral profiles that align with actual in-game mechanics and player behavior?
• RQ2 How do k-means, NMF, PCA, and AA differ in their ability to represent extreme or archetypical player behaviors rather than central tendencies?
• RQ3 To what extent do PCA and NMF generate basis vectors that imply infeasible behaviors (e.g., character level loss) in a game like World of Warcraft?
• RQ4 How does the distribution of players across behavioral clusters differ between hard-assignment (k-means) and soft-assignment (NMF, PCA, AA) methods?
• RQ5 Can Archetypal Analysis via SIVM produce behaviorally valid, distinct, and actionable profiles for game developers?

Key findings

• Archetypal Analysis (AA) produced basis vectors that correspond to actual, legal player behaviors—such as rapid level progression after expansion releases—while PCA and NMF generated profiles implying players could lose levels, which is impossible in World of Warcraft.
• k-means centroids were interpretable but overall very similar in shape, limiting their ability to distinguish distinct behavioral types.
• AA basis vectors were significantly more varied and interpretable than k-means centroids, enabling clearer behavioral labeling and design insights.
• The player distribution across clusters varied substantially: AA and PCA indicated three large groups, while k-means and NMF showed four large and four smaller groups.
• Only k-means fully supported hard cluster assignments, enabling precise quantification of player counts per behavioral type, whereas other methods required approximation via nearest-basis assignment.
• AA demonstrated superior interpretability and behavioral validity, making it a promising method for generating actionable player profiles in game analytics.