[Paper Review] Metrics for Multi-Class Classification: an Overview

Margherita Grandini, E. Bagli+1 more|arXiv (Cornell University)|Aug 13, 2020

Machine Learning in Bioinformatics9 references66 citations

TL;DR

This white paper surveys multi-class classification performance metrics, detailing their definitions, advantages, disadvantages, and use cases for model evaluation and development.

ABSTRACT

Classification tasks in machine learning involving more than two classes are known by the name of "multi-class classification". Performance indicators are very useful when the aim is to evaluate and compare different classification models or machine learning techniques. Many metrics come in handy to test the ability of a multi-class classifier. Those metrics turn out to be useful at different stage of the development process, e.g. comparing the performance of two different models or analysing the behaviour of the same model by tuning different parameters. In this white paper we review a list of the most promising multi-class metrics, we highlight their advantages and disadvantages and show their possible usages during the development of a classification model.

Motivation & Objective

• Review a broad set of promising multi-class evaluation metrics.
• Highlight the advantages and disadvantages of each metric.
• Show how metrics can be used during the development and tuning of classification models.
• Explain how class balance affects metric choice and interpretation.

Proposed method

• Ground the discussion in confusion-matrix based metrics and their extensions.
• Present and define Accuracy, Balanced Accuracy, and Balanced Accuracy Weighted.
• Explain F1-Score in binary and multi-class settings, including Macro and Micro variants.
• Introduce Cross-Entropy as a distribution-based evaluation measure.
• Discuss independence-based metrics (Matthews Correlation Coefficient and Cohen’s Kappa) and their multi-class extensions.
• Provide practical interpretations, pros/cons, and guidance for metric selection.

Experimental results

Research questions

• RQ1 What are the key metrics for evaluating multi-class classifiers and how do they differ in interpretation?
• RQ2 How do macro- and micro-averaging affect the assessment of multi-class performance, especially under class imbalance?
• RQ3 When are alternative metrics like MCC and Cohen’s Kappa preferable to standard accuracy or F1 scores?
• RQ4 How does cross-entropy relate to classification performance and what are its limitations in evaluation?

Key findings

• Accuracy is a dataset-level measure summing correctly and incorrectly classified units.
• Balanced Accuracy averages per-class recall, mitigating class-imbalance effects.
• Balanced Accuracy Weighted incorporates class frequency to balance per-class recalls.
• F1-Score combines Precision and Recall via harmonic mean, with Macro and Micro variants for multi-class evaluation.
• Macro F1 treats all classes equally, while Micro F1 aligns with overall Accuracy; each has distinct interpretation.
• Cross-Entropy evaluates distribution similarity rather than the quality of the decision rule and is computationally fast.
• Matthews Correlation Coefficient and Cohen’s Kappa extend dependence/ concordance concepts to multi-class, accounting for TNs and chance agreement; they offer balanced assessments and can outperform simple accuracy in unbalanced settings.