[Paper Review] Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology

Alexei Botchkarev|arXiv (Cornell University)|Sep 9, 2018

Forecasting Techniques and Applications28 references386 citations

TL;DR

This paper overviews performance metrics (error measures) across machine learning regression, forecasting, and prognostics and proposes a typology to aid metric selection. It introduces a four-category framework and identifies key components shaping primary metrics.

ABSTRACT

Performance metrics (error measures) are vital components of the evaluation frameworks in various fields. The intention of this study was to overview of a variety of performance metrics and approaches to their classification. The main goal of the study was to develop a typology that will help to improve our knowledge and understanding of metrics and facilitate their selection in machine learning regression, forecasting and prognostics. Based on the analysis of the structure of numerous performance metrics, we propose a framework of metrics which includes four (4) categories: primary metrics, extended metrics, composite metrics, and hybrid sets of metrics. The paper identified three (3) key components (dimensions) that determine the structure and properties of primary metrics: method of determining point distance, method of normalization, method of aggregation of point distances over a data set.

Motivation & Objective

• Motivate the need for a comprehensive overview of error measures across regression, forecasting, and prognostics.
• Develop a typology to improve understanding and selection of metrics in these domains.
• Analyze the structure of performance metrics to identify governing dimensions and properties.

Proposed method

• Analyze existing performance metrics from the literature and practice.
• Identify structural patterns and classifications within metrics.
• Propose a framework with four metric categories: primary metrics, extended metrics, composite metrics, and hybrid sets.
• Delineate three key dimensions that determine primary metric properties: distance/dissimilarity calculation method, normalization method, and aggregation method over a data set.

Experimental results

Research questions

• RQ1 What are the major categories and structures of performance metrics used in regression, forecasting, and prognostics?
• RQ2 How can metrics be classified to improve selection and understanding in ML evaluation?
• RQ3 What are the fundamental dimensions that determine the properties of primary error metrics?

Key findings

• A four-category typology of metrics is proposed: primary metrics, extended metrics, composite metrics, and hybrid metric sets.
• Three dimensions determine primary metric structure: how point distances are computed, how distances are normalized, and how distances are aggregated over data sets.
• The framework aims to facilitate better knowledge, interpretation, and selection of metrics in ML evaluation.
• The study provides an organizing perspective to compare and contrast different error measures across domains.