[Paper Review] HyperSTAR: Task-Aware Hyperparameters for Deep Networks

Gaurav Mittal, Chang Liu+4 more|arXiv (Cornell University)|May 21, 2020

Advanced Neural Network Applications47 references4 citations

TL;DR

HyperSTAR is a task-aware hyperparameter optimization framework that learns a joint representation of datasets and hyperparameters from raw images to recommend high-performing configurations. By end-to-end training a model to predict performance on unseen tasks, HyperSTAR reduces HPO search time by 50% and enables Hyperband to achieve optimal accuracy in just 25% of the budget required by vanilla or Bayesian-optimized variants.

ABSTRACT

While deep neural networks excel in solving visual recognition tasks, they require significant effort to find hyperparameters that make them work optimally. Hyperparameter Optimization (HPO) approaches have automated the process of finding good hyperparameters but they do not adapt to a given task (task-agnostic), making them computationally inefficient. To reduce HPO time, we present HyperSTAR (System for Task Aware Hyperparameter Recommendation), a task-aware method to warm-start HPO for deep neural networks. HyperSTAR ranks and recommends hyperparameters by predicting their performance conditioned on a joint dataset-hyperparameter space. It learns a dataset (task) representation along with the performance predictor directly from raw images in an end-to-end fashion. The recommendations, when integrated with an existing HPO method, make it task-aware and significantly reduce the time to achieve optimal performance. We conduct extensive experiments on 10 publicly available large-scale image classification datasets over two different network architectures, validating that HyperSTAR evaluates 50% less configurations to achieve the best performance compared to existing methods. We further demonstrate that HyperSTAR makes Hyperband (HB) task-aware, achieving the optimal accuracy in just 25% of the budget required by both vanilla HB and Bayesian Optimized HB~(BOHB).

Motivation & Objective

• To address the inefficiency of task-agnostic hyperparameter optimization (HPO) methods that treat all datasets equally, regardless of their visual characteristics.
• To develop a method that leverages visual priors from raw images to learn a joint dataset-hyperparameter representation for better HPO efficiency.
• To enable warm-starting of existing HPO algorithms by recommending high-performing hyperparameter configurations tailored to the specific task at hand.
• To significantly reduce the number of configurations evaluated while achieving optimal performance, especially under strict computational budgets.

Proposed method

• HyperSTAR learns a task representation directly from raw training images using a deep neural network in an end-to-end fashion.
• It jointly trains a performance predictor that estimates accuracy for a given hyperparameter configuration based on the learned task representation.
• The method uses a meta-learning framework with two phases: offline meta-learning over a diverse set of past datasets and online recommendation for new, unseen datasets.
• HyperSTAR ranks hyperparameter configurations by predicted performance, enabling prioritization in HPO pipelines such as Hyperband.
• The framework integrates with existing HPO methods by replacing their random or uniform configuration sampling with a task-aware ranking of candidates.
• It leverages a contrastive learning objective to align representations of similar datasets and improve generalization across tasks.

Experimental results

Research questions

• RQ1 Can a model learn to predict optimal hyperparameter configurations for a new image classification task based solely on raw images, without relying on hand-crafted meta-features?
• RQ2 How effective is a joint representation of datasets and hyperparameters in improving the efficiency of hyperparameter optimization?
• RQ3 To what extent can HyperSTAR reduce the number of configurations evaluated while maintaining or improving final model accuracy?
• RQ4 Can HyperSTAR be used to accelerate state-of-the-art HPO algorithms like Hyperband, especially under low-budget settings?

Key findings

• HyperSTAR reduces the number of evaluated hyperparameter configurations by 50% compared to state-of-the-art methods while achieving the same optimal Top-1 accuracy on 10 large-scale image classification datasets.
• When integrated with Hyperband, HyperSTAR achieves optimal performance in just 25% of the budget required by vanilla Hyperband and Bayesian-optimized Hyperband (BOHB).
• For the smallest budget setting (100 epochs), HyperSTAR-based Hyperband achieves an average of 1.5% higher Top-1 accuracy than vanilla HB and BOHB.
• The method outperforms Tr-AutoML in efficiency, achieving comparable performance in 10 times less evaluation budget by learning from raw pixels rather than relying on dataset grouping.
• The performance gap between HyperSTAR and baseline methods diminishes with increasing budget, indicating that HyperSTAR is both efficient and effective even under large-budget conditions.
• The ablation studies confirm that end-to-end learning of the joint dataset-hyperparameter representation is critical for strong generalization and recommendation quality.