[Paper Review] PyTorch Hyperparameter Tuning - A Tutorial for spotPython

Thomas Bartz–Beielstein|arXiv (Cornell University)|May 19, 2023

Computational Physics and Python Applications4 citations

TL;DR

This tutorial introduces spotPython, a Python-based hyperparameter tuning framework inspired by the R-based SPOT toolbox, for optimizing PyTorch deep learning models. It demonstrates integration into a CIFAR10 image classification workflow using surrogate model-based optimization, showing comparable or better performance than Ray Tune with improved transparency and flexibility.

ABSTRACT

The goal of hyperparameter tuning (or hyperparameter optimization) is to optimize the hyperparameters to improve the performance of the machine or deep learning model. spotPython (``Sequential Parameter Optimization Toolbox in Python'') is the Python version of the well-known hyperparameter tuner SPOT, which has been developed in the R programming environment for statistical analysis for over a decade. PyTorch is an optimized tensor library for deep learning using GPUs and CPUs. This document shows how to integrate the spotPython hyperparameter tuner into the PyTorch training workflow. As an example, the results of the CIFAR10 image classifier are used. In addition to an introduction to spotPython, this tutorial also includes a brief comparison with Ray Tune, a Python library for running experiments and tuning hyperparameters. This comparison is based on the PyTorch hyperparameter tuning tutorial. The advantages and disadvantages of both approaches are discussed. We show that spotPython achieves similar or even better results while being more flexible and transparent than Ray Tune.

Motivation & Objective

• To enable efficient and interpretable hyperparameter tuning for PyTorch models using spotPython.
• To address the challenge of limited computational resources and the need for model transparency in hyperparameter optimization.
• To provide a practical, reproducible workflow integrating spotPython into standard PyTorch training pipelines.
• To compare spotPython with Ray Tune in terms of performance, flexibility, and interpretability in hyperparameter tuning.

Proposed method

• Utilizes surrogate model-based optimization with Gaussian process (Kriging) models for efficient hyperparameter search.
• Integrates spotPython into PyTorch workflows via modular components: data loading, preprocessing (prep_model), model definition (core_model), and training/evaluation loops.
• Employs cross-validation and early stopping during training to improve generalization and reduce overfitting.
• Supports both k-fold cross-validation and hold-out validation settings for performance evaluation.
• Uses statistical tools like sensitivity analysis and fitness landscape exploration to interpret hyperparameter effects.
• Enables model interpretability by quantifying contributions of hyperparameters and their interactions through statistical modeling.

Experimental results

Research questions

• RQ1 How can spotPython be effectively integrated into a standard PyTorch training pipeline for hyperparameter tuning?
• RQ2 What performance advantages does spotPython offer over Ray Tune in terms of model accuracy and convergence speed?
• RQ3 To what extent does spotPython improve transparency and interpretability in hyperparameter optimization compared to black-box methods?
• RQ4 How does surrogate model-based optimization in spotPython handle limited computational resources and noisy evaluations?
• RQ5 Can spotPython achieve comparable or better results than Ray Tune while maintaining greater configurability and auditability?

Key findings

• spotPython achieves similar or better performance than Ray Tune in hyperparameter tuning for the CIFAR10 image classifier, with improved convergence and stability.
• The surrogate model-based approach in spotPython enables efficient exploration of the hyperparameter space with fewer evaluations, reducing computational cost.
• spotPython provides greater transparency and interpretability by quantifying the impact of individual hyperparameters and their interactions through statistical analysis.
• The integration of spotPython into PyTorch workflows is seamless and modular, supporting both cross-validation and hold-out evaluation settings.
• The framework supports early stopping and model checkpointing, enhancing training efficiency and preventing overfitting.
• The open-source nature of spotPython ensures reproducibility and auditability, with the code available on GitHub for public use and extension.