[Paper Review] Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification

Bo Yang Hsueh, Wei Li+1 more|arXiv (Cornell University)|Jun 5, 2018

Advanced Neural Network Applications23 references4 citations

TL;DR

This paper proposes Hyperbolic-Tangent Decay (HTD), a novel learning rate scheduler for stochastic gradient descent (SGD) that uses a hyperbolic-tangent function to dynamically adjust the learning rate during training. HTD outperforms step decay and cosine schedulers across multiple benchmarks—including ResNet, Wide ResNet, DenseNet, and LSTM—while requiring fewer hyperparameters than step decay and offering greater flexibility than cosine scheduling, with HTD(-6,3) emerging as a robust default configuration.

ABSTRACT

Learning rate scheduler has been a critical issue in the deep neural network training. Several schedulers and methods have been proposed, including step decay scheduler, adaptive method, cosine scheduler and cyclical scheduler. This paper proposes a new scheduling method, named hyperbolic-tangent decay (HTD). We run experiments on several benchmarks such as: ResNet, Wide ResNet and DenseNet for CIFAR-10 and CIFAR-100 datasets, LSTM for PAMAP2 dataset, ResNet on ImageNet and Fashion-MNIST datasets. In our experiments, HTD outperforms step decay and cosine scheduler in nearly all cases, while requiring less hyperparameters than step decay, and more flexible than cosine scheduler. Code is available at https://github.com/BIGBALLON/HTD.

Motivation & Objective

• To address the challenge of hyperparameter sensitivity and suboptimal performance in existing learning rate schedulers for SGD in deep learning.
• To develop a learning rate scheduler that combines the performance benefits of step decay with the flexibility of adaptive methods while reducing the number of tunable parameters.
• To evaluate the effectiveness of a new scheduler based on the hyperbolic-tangent function across diverse architectures and datasets.
• To identify a robust, default configuration for the new scheduler that generalizes well across different training scenarios.

Proposed method

• The proposed HTD scheduler uses a hyperbolic-tangent function to model the learning rate decay over training epochs, defined as $\alpha_t = \frac{L - U}{2} \cdot \tanh\left(\frac{t}{R}\right) + \frac{L + U}{2}$, where $L$ and $U$ are the lower and upper bounds of the learning rate, and $R$ controls the decay ratio.
• The learning rate starts near $U$ at the beginning of training and asymptotically approaches $L$ as training progresses, enabling smooth and gradual decay.
• The scheduler is designed to be more flexible than cosine decay (which only allows tuning of min and max rates) and less parameter-intensive than step decay (which requires multiple stage boundaries).
• The method is evaluated using standard deep learning benchmarks: CIFAR-10, CIFAR-100, ImageNet, PAMAP2, and Fashion-MNIST, with models including ResNet, Wide ResNet, DenseNet, and BLSTM.
• Hyperparameter sensitivity is analyzed by varying $L$, $U$, and $R$ across different datasets and training epochs to assess robustness and generalization.

Experimental results

Research questions

• RQ1 Can a learning rate scheduler based on the hyperbolic-tangent function outperform established schedulers like step decay and cosine decay in terms of final model accuracy?
• RQ2 Does HTD require fewer hyperparameters than step decay while maintaining or improving performance?
• RQ3 How does the flexibility of HTD compare to cosine scheduling, particularly in terms of tunable parameters and final accuracy?
• RQ4 What is the optimal configuration of HTD hyperparameters (e.g., $L$, $U$, $R$) across different datasets and model architectures?
• RQ5 Does the performance of HTD vary significantly with training duration, and can a single configuration generalize across different training schedules?

Key findings

• HTD(-6,3) outperforms step decay in all evaluated experiments, achieving a 0.58% lower top-1 error rate on ImageNet with ResNet-18 and a 0.48% lower error with ResNet-34.
• On CIFAR-100, HTD(-6,3) improves accuracy by 0.42% over the cosine scheduler for both DenseNet-BC-100-12 and DenseNet-BC-250-24.
• For WRN-28-10 on CIFAR-100, HTD(-6,3) achieves a 0.81% improvement over the cosine scheduler.
• On PAMAP2, HTD(-6,3) reduces the error rate by 0.16% compared to step decay and by 0.22% compared to the cosine scheduler.
• On Fashion-MNIST, HTD(-6,3) improves performance by 0.15% over step decay and 0.06% over the cosine scheduler.
• The optimal ratio $R$ varies by dataset: $R=5$ for CIFAR-10 with 200 epochs and $R=2$ with 400 epochs, while $R=10$ is optimal for CIFAR-100 across both settings, indicating dataset-dependent sensitivity.