[Paper Review] Unsure When to Stop? Ask Your Semantic Neighbors

Ivo Gonçalves, Sara Silva+2 more|arXiv (Cornell University)|Jun 19, 2017

Evolutionary Algorithms and Applications10 references3 citations

TL;DR

This paper proposes semantic stopping criteria that leverage the semantic neighborhood of solutions to determine optimal termination points in Geometric Semantic Genetic Programming (GSGP) and the Semantic Learning Machine (SLM), significantly improving generalization while reducing computational cost. The method uses semantic similarity and incremental evaluation to halt evolution early, achieving competitive performance in under 3 seconds for neural networks and under 10 seconds for GP trees.

ABSTRACT

In iterative supervised learning algorithms it is common to reach a point in the search where no further induction seems to be possible with the available data. If the search is continued beyond this point, the risk of overfitting increases significantly. Following the recent developments in inductive semantic stochastic methods, this paper studies the feasibility of using information gathered from the semantic neighborhood to decide when to stop the search. Two semantic stopping criteria are proposed and experimentally assessed in Geometric Semantic Genetic Programming (GSGP) and in the Semantic Learning Machine (SLM) algorithm (the equivalent algorithm for neural networks). The experiments are performed on real-world high-dimensional regression datasets. The results show that the proposed semantic stopping criteria are able to detect stopping points that result in a competitive generalization for both GSGP and SLM. This approach also yields computationally efficient algorithms as it allows the evolution of neural networks in less than 3 seconds on average, and of GP trees in at most 10 seconds. The usage of the proposed semantic stopping criteria in conjunction with the computation of optimal mutation/learning steps also results in small trees and neural networks.

Motivation & Objective

• To address overfitting in iterative supervised learning by identifying optimal stopping points before performance degrades on unseen data.
• To explore whether semantic neighborhood information—derived from geometric semantic operators—can guide effective and efficient stopping decisions.
• To evaluate the feasibility and performance of semantic stopping criteria in both GSGP and SLM, two algorithms rooted in geometric semantic theory.
• To assess the impact of combining semantic stopping with optimal mutation/learning step computation on model size and generalization.
• To demonstrate computational efficiency, enabling fast evolution of compact models without sacrificing predictive performance.

Proposed method

• Proposes two semantic stopping criteria: one based on semantic similarity to the target (Semantic Similarity-based Stopping, SSS), and another based on the rate of semantic improvement (Semantic Improvement-based Stopping, SIS).
• Applies these criteria within the GSGP and SLM frameworks, using geometric semantic operators to maintain unimodal fitness landscapes and ensure convergence toward the global optimum.
• Employs incremental evaluation at each mutation or learning step to efficiently track semantic changes and avoid full re-evaluation.
• Introduces the computation of optimal mutation and learning steps to further enhance convergence speed and model compactness.
• Uses semantic space distance metrics to quantify how close a solution is to the target, enabling early detection of stagnation or overfitting.
• Applies the stopping criteria dynamically during evolution, halting the process when semantic progress falls below a threshold or when similarity to the target plateaus.

Experimental results

Research questions

• RQ1 Can semantic neighborhood information reliably signal when further evolution will lead to overfitting in GSGP and SLM?
• RQ2 Do the proposed semantic stopping criteria yield models with better generalization performance compared to standard stopping heuristics?
• RQ3 To what extent does the integration of semantic stopping with optimal step computation reduce model size and training time?
• RQ4 How efficient are the resulting algorithms in terms of computational time, especially on high-dimensional regression datasets?
• RQ5 Is the proposed approach generalizable across different types of supervised learning problems, including both regression and classification?

Key findings

• The proposed semantic stopping criteria achieve competitive generalization performance on real-world high-dimensional regression datasets, outperforming or matching standard stopping strategies.
• On average, neural network evolution with SLM completes in less than 3 seconds per run, with the OS EDV variant achieving a median of 0.3 seconds on the LD50 dataset.
• GP tree evolution with SSHC completes in at most 10 seconds per run, with the OS EDV variant reaching a median of 0.9 seconds on the Bio dataset.
• The use of semantic stopping in conjunction with optimal mutation steps results in significantly smaller models: GP trees have a median of 327 nodes (OS EDV) versus over 10,000 in non-optimized variants.
• The SLM variant using FS TIE with semantic stopping achieves the best generalization on two out of three datasets (PPB and LD50), despite larger network sizes.
• The OS EDV variant of SLM produces the smallest neural networks, with a median of 4 hidden neurons across datasets, while maintaining strong generalization performance.