[Paper Review] Testing the Efficient Network TRaining (ENTR) Hypothesis: initially reducing training image size makes Convolutional Neural Network training for image recognition tasks more efficient

Thomas Cherico Wanger, Peter Frohn-Sörensen|arXiv (Cornell University)|Jul 30, 2018

Advanced Neural Network Applications19 references3 citations

TL;DR

This paper tests the Efficient Network Training (ENTR) hypothesis, proposing that training Convolutional Neural Networks (CNNs) on smaller images initially, then gradually increasing size, improves training efficiency. Experiments on ResNet models across three small datasets show consistent accuracy gains and reduced inference time, attributed to implicit regularization from early small-scale training.

ABSTRACT

Convolutional Neural Networks (CNN) for image recognition tasks are seeing rapid advances in the available architectures and how networks are trained based on large computational infrastructure and standard datasets with millions of images. In contrast, performance and time constraints for example, of small devices and free cloud GPUs necessitate efficient network training (i.e., highest accuracy in the shortest inference time possible), often on small datasets. Here, we hypothesize that initially decreasing image size during training makes the training process more efficient, because pre-shaping weights with small images and later utilizing these weights with larger images reduces initial network parameters and total inference time. We test this Efficient Network TRaining (ENTR) Hypothesis by training pre-trained Residual Network (ResNet) models (ResNet18, 34, & 50) on three small datasets (steel microstructures, bee images, and geographic aerial images) with a free cloud GPU. Based on three training regimes of i) not, ii) gradually or iii) in one step increasing image size over the training process, we show that initially reducing image size increases training efficiency consistently across datasets and networks. We interpret these results mechanistically in the framework of regularization theory. Support for the ENTR hypothesis is an important contribution, because network efficiency improvements for image recognition tasks are needed for practical applications. In the future, it will be exciting to see how the ENTR hypothesis holds for large standard datasets like ImageNet or CIFAR, to better understand the underlying mechanisms, and how these results compare to other fields such as structural learning.

Motivation & Objective

• To investigate whether reducing initial training image size improves training efficiency for image recognition tasks.
• To evaluate the impact of image size scheduling on model accuracy and inference time across diverse small datasets.
• To explore the mechanistic basis of improved efficiency through regularization theory.
• To assess the feasibility and benefits of ENTR for resource-constrained environments like small devices and free cloud GPUs.
• To provide empirical validation of the ENTR hypothesis beyond standard large-scale datasets.

Proposed method

• Training pre-trained ResNet18, ResNet34, and ResNet50 models on three small datasets: steel microstructures, bee images, and aerial geospatial images.
• Implementing three training regimes: (i) no size change, (ii) gradual increase in image size, and (iii) one-step increase from small to large size.
• Using Google Colaboratory’s free cloud GPU to simulate resource-constrained training conditions.
• Measuring training efficiency via accuracy, convergence speed, and total inference time.
• Analyzing results through the lens of regularization theory to explain observed improvements.
• Comparing performance across different image size schedules to isolate the effect of initial small-scale training.

Experimental results

Research questions

• RQ1 Does training CNNs on smaller images initially lead to more efficient training in terms of accuracy and inference time?
• RQ2 How does the image size scheduling strategy (gradual vs. one-step increase) affect model convergence and performance?
• RQ3 What is the underlying mechanism explaining the observed efficiency gains in the ENTR approach?
• RQ4 How does the ENTR hypothesis perform across diverse small datasets in material science, geoscience, and environmental science?
• RQ5 Can the ENTR approach be generalized to standard datasets like ImageNet or CIFAR in future work?

Key findings

• The ENTR approach—training on smaller images initially and then increasing size—consistently improved training efficiency across all tested ResNet architectures and datasets.
• Models trained with initial small image sizes achieved higher accuracy with significantly reduced inference time compared to full-size training.
• The gradual increase in image size outperformed the one-step increase, suggesting a smoother optimization trajectory.
• The improvement was attributed to implicit regularization, where early small-scale training stabilizes learning and reduces overfitting.
• The method proved effective even on free cloud GPUs, demonstrating practical viability for low-resource environments.
• The results support the hypothesis that initial small-image training acts as a form of inductive bias that enhances generalization and efficiency.