[Paper Review] EPP: interpretable score of model predictive power

Alicja Gosiewska, Mateusz Bakala+3 more|arXiv (Cornell University)|Aug 24, 2019

Machine Learning and Data Classification8 references4 citations

TL;DR

This paper introduces EPP (Elo-based Predictive Power), a novel, interpretable scoring system for evaluating machine learning model performance. EPP transforms model evaluation by enabling probabilistic interpretation of performance differences, direct cross-dataset comparisons, and stability assessment in cross-validation, offering a robust alternative to traditional metrics like AUC or RMSE.

ABSTRACT

The most important part of model selection and hyperparameter tuning is the evaluation of model performance. The most popular measures, such as AUC, F1, ACC for binary classification, or RMSE, MAD for regression, or cross-entropy for multilabel classification share two common weaknesses. First is, that they are not on an interval scale. It means that the difference in performance for the two models has no direct interpretation. It makes no sense to compare such differences between datasets. Second is, that for k-fold cross-validation, the model performance is in most cases calculated as an average performance from particular folds, which neglects the information how stable is the performance for different folds. In this talk, we introduce a new EPP rating system for predictive models. We also demonstrate numerous advantages for this system, First, differences in EPP scores have probabilistic interpretation. Based on it we can assess the probability that one model will achieve better performance than another. Second, EPP scores can be directly compared between datasets. Third, they can be used for navigated hyperparameter tuning and model selection. Forth, we can create embeddings for datasets based on EPP scores.

Motivation & Objective

• Address the lack of interpretable, interval-scale performance metrics in machine learning model evaluation.
• Overcome the limitations of common metrics like AUC, F1, and RMSE, which lack probabilistic interpretation and cannot be meaningfully compared across datasets.
• Provide a method to assess model performance stability across cross-validation folds.
• Enable navigated hyperparameter tuning by quantifying the probability of improvement through further search.
• Support the creation of data set embeddings based on EPP scores for downstream model selection and analysis.

Proposed method

• Adapt the Elo rating system—originally used for ranking players in games—into a predictive performance scoring framework for machine learning models.
• Define EPP scores based on pairwise comparisons of model performances across multiple train/test splits, treating each comparison as a 'match' where the better-performing model wins.
• Use a modified Elo update rule to compute EPP scores, where the expected outcome is derived from the performance difference between models.
• Apply the EPP system to model comparisons across multiple data sets and hyperparameter configurations, generating a consistent, interpretable scale.
• Utilize EPP scores to create embeddings of data sets via dimensionality reduction (e.g., PCA), revealing structural relationships between data sets and model behaviors.
• Integrate EPP into hyperparameter tuning workflows by estimating the probability of further performance gains, enabling intelligent search termination and navigation.

Experimental results

Research questions

• RQ1 Can a performance metric be developed that provides a probabilistic interpretation of model performance differences across models and datasets?
• RQ2 To what extent can EPP scores enable meaningful cross-dataset comparisons of model performance, despite differing data scales and label distributions?
• RQ3 How can EPP scores be used to assess the stability of model performance across k-fold cross-validation folds?
• RQ4 Can EPP scores support navigated hyperparameter tuning by estimating the likelihood of further improvement?
• RQ5 Can EPP-based embeddings reveal meaningful structure in data sets, such as similarity in model behavior or difficulty?

Key findings

• EPP scores provide a probabilistic interpretation of performance differences: the difference between two EPP scores allows estimation of the probability that one model will outperform another.
• EPP scores are directly comparable across datasets, resolving the issue of incommensurability in traditional metrics like AUC or RMSE.
• EPP scores reveal performance instability in cross-validation: models with highly variable fold performances show lower EPP scores, indicating less reliable ranking.
• EPP enables navigated hyperparameter tuning: by analyzing EPP trends across hyperparameter settings, one can identify promising regions and estimate the probability of further improvement.
• EPP-based PCA biplots reveal structural relationships between data sets—e.g., data set 334 showed reversed model ranking patterns compared to others, indicating unique characteristics.
• Random forest and GBM models consistently achieved positive EPP scores across data sets, indicating they generally outperform the average model, while k-NN performance varied significantly by data set.