[Paper Review] On Metrics to Assess the Transferability of Machine Learning Models in Non-Intrusive Load Monitoring

Christoph Klemenjak, Anthony Faustine+2 more|arXiv (Cornell University)|Dec 12, 2019

Smart Grid Energy Management12 citations

TL;DR

This paper introduces four domain-specific metrics—Generalisation Ratio (GR), Accuracy on Unseen Houses (AUH), Error on Unseen Houses (EUH), and Mean Generalisation Loss (MGL)—to evaluate the transferability of machine learning models in Non-Intrusive Load Monitoring (NILM). The metrics assess model performance on unseen buildings, enabling detection of overfitting and complementing traditional metrics, with validation across five state-of-the-art and four baseline NILM algorithms on multiple datasets.

ABSTRACT

To assess the performance of load disaggregation algorithms it is common practise to train a candidate algorithm on data from one or multiple households and subsequently apply cross-validation by evaluating the classification and energy estimation performance on unseen portions of the dataset derived from the same households. With an emerging discussion of transferability in Non-Intrusive Load Monitoring (NILM), there is a need for domain-specific metrics to assess the performance of NILM algorithms on new test scenarios being unseen buildings. In this paper, we discuss several metrics to assess the generalisation ability of NILM algorithms. These metrics target different aspects of performance evaluation in NILM and are meant to complement the traditional performance evaluation approach. We demonstrate how our metrics can be utilised to evaluate NILM algorithms by means of two case studies. We conduct our studies on several energy consumption datasets and take into consideration five state-of-the-art as well as four baseline NILM solutions. Finally, we formulate research challenges for future work.

Motivation & Objective

• Address the lack of domain-specific metrics to evaluate transferability of NILM models across unseen buildings.
• Provide a systematic evaluation framework to assess generalization performance of NILM algorithms beyond seen households.
• Enable researchers to detect overfitting by quantifying performance on unseen test scenarios.
• Complement existing performance metrics with transferability-aware evaluation to improve reproducibility and model selection.
• Formulate open challenges for future research in NILM generalization and transfer learning.

Proposed method

• Propose Generalisation Ratio (GR) as the ratio of performance on unseen houses to performance on seen houses, measuring relative transferability.
• Introduce Accuracy on Unseen Houses (AUH) and Error on Unseen Houses (EUH) to quantify classification and regression performance on new buildings.
• Define Mean Generalisation Loss (MGL) as the average loss on unseen houses, applicable only in 1-to-N or 1-to-1 transfer settings.
• Apply metrics across multiple datasets and evaluate five state-of-the-art and four baseline NILM algorithms using both classification and regression tasks.
• Use case studies to demonstrate metric utility in real-world evaluation scenarios, including model comparison and overfitting detection.
• Provide open-source implementation on GitHub to ensure reproducibility and facilitate adoption by future researchers.

Experimental results

Research questions

• RQ1 How can the generalization performance of NILM models be quantitatively assessed when tested on unseen buildings?
• RQ2 What metrics best capture the transferability of NILM models across different households or datasets?
• RQ3 How do traditional performance metrics fail to reflect model generalization, and what improvements do the proposed metrics offer?
• RQ4 To what extent can the proposed metrics detect overfitting in NILM models trained on specific households?
• RQ5 How can transferability be evaluated consistently across diverse NILM evaluation scenarios, including 1-to-N and M-to-N settings?

Key findings

• The Generalisation Ratio (GR) effectively captures relative performance degradation when transferring models to unseen buildings, with lower values indicating poorer transferability.
• Accuracy on Unseen Houses (AUH) and Error on Unseen Houses (EUH) provide direct, interpretable measures of model performance on new test scenarios, enabling cross-algorithm comparison.
• Mean Generalisation Loss (MGL) is a reliable metric for 1-to-N transfer scenarios but requires adaptation for M-to-N training settings.
• The proposed metrics successfully identify overfitting in NILM models, especially when performance on seen houses is high but performance on unseen houses is poor.
• Case studies demonstrate that state-of-the-art models do not always generalize well, and the use of transferability metrics reveals significant performance drops on unseen buildings.
• The authors recommend integrating GR and AUH/EUH into standard NILM evaluation pipelines to improve model robustness and reproducibility.