[Paper Review] Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputs

Emmanuel Abbé, Elisabetta Cornacchia+1 more|arXiv (Cornell University)|Jun 29, 2023

Neural Networks and Applications4 citations

TL;DR

This paper proves a theoretical advantage of curriculum learning (CL) over standard training for learning high-degree parities using ReLU networks with mixed sparse and dense inputs. It shows that a 2-layer ReLU network trained with curriculum (starting on sparse inputs) learns any bounded-degree parity in θ(d) steps, while standard training requires Ω(d²) steps under the same conditions, establishing a provable separation in sample efficiency.

ABSTRACT

Experimental results have shown that curriculum learning, i.e., presenting simpler examples before more complex ones, can improve the efficiency of learning. Some recent theoretical results also showed that changing the sampling distribution can help neural networks learn parities, with formal results only for large learning rates and one-step arguments. Here we show a separation result in the number of training steps with standard (bounded) learning rates on a common sample distribution: if the data distribution is a mixture of sparse and dense inputs, there exists a regime in which a 2-layer ReLU neural network trained by a curriculum noisy-GD (or SGD) algorithm that uses sparse examples first, can learn parities of sufficiently large degree, while any fully connected neural network of possibly larger width or depth trained by noisy-GD on the unordered samples cannot learn without additional steps. We also provide experimental results supporting the qualitative separation beyond the specific regime of the theoretical results.

Motivation & Objective

• To establish a provable separation between curriculum learning and standard training in learning parities on mixed input distributions.
• To analyze the role of sparse inputs in enabling efficient learning of high-degree parities via gradient-based methods.
• To demonstrate that curriculum training with sparse-first sampling reduces the number of steps required for convergence compared to random-order training.
• To extend theoretical results beyond idealized assumptions to validate the separation empirically across multiple architectures (MLP, mean-field, Transformer).
• To investigate whether the curriculum advantage generalizes to other large-leap functions beyond parities.

Proposed method

• The authors analyze a 2-layer ReLU fully connected network trained via noisy gradient descent (GD) or SGD on a mixed input distribution with fraction ρ of sparse inputs.
• They define curriculum learning as an initial phase where only sparse inputs are used, followed by training on the full mixed dataset.
• Theoretical analysis uses bounded learning rates and focuses on the number of training steps required to learn k-parity functions.
• The key theoretical result shows that with ρ < d⁻⁴, curriculum training achieves convergence in θ(d) steps, while standard training requires Ω(d²) steps.
• Experiments validate the theoretical separation across multiple models: MLP, mean-field, and Transformer, using ℓ₂, hinge, and covariance losses.
• The experiments vary ρ, parity degree k, and dataset size to measure gains in sample complexity and optimization steps.

Experimental results

Research questions

• RQ1 Can curriculum learning provably reduce the number of training steps required to learn high-degree parities compared to standard training on a common mixed input distribution?
• RQ2 What is the role of sparse inputs in enabling faster convergence for parity learning in ReLU networks?
• RQ3 Does the theoretical advantage of curriculum learning hold beyond idealized assumptions, such as layerwise training and specific learning rates?
• RQ4 Can the curriculum advantage be observed across different neural network architectures and loss functions?
• RQ5 Is the curriculum effect robust to changes in the fraction of sparse inputs (ρ) and parity degree (k)?

Key findings

• For ρ < d⁻⁴, a 2-layer ReLU network trained with curriculum learning learns any k-parity in θ(d) training steps.
• The same network trained with standard random-order sampling requires Ω(d²) steps, establishing a provable Ω(d) separation in sample efficiency.
• Experiments on MLPs show that curriculum training reduces the number of steps needed for convergence, especially for higher-degree parities and smaller ρ.
• The mean-field model also exhibits a significant reduction in steps and sample complexity when using curriculum, with gains visible across ℓ₂, hinge, and covariance losses.
• The Transformer model shows a weaker but still significant advantage under curriculum learning, particularly for small ρ (e.g., 0.001), indicating broader applicability.
• The results suggest that curriculum learning is particularly effective when sparse inputs are rare but informative, enabling early identification of the parity support.