[Paper Review] Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation

David Bonet, Antonio Ortega+2 more|arXiv (Cornell University)|Jul 27, 2021

Domain Adaptation and Few-Shot Learning25 references4 citations

TL;DR

This paper proposes Channel-Wise DeepNNK (CW-DeepNNK), a novel early stopping criterion for ConvNets that estimates generalization performance via local polytope interpolation on individual network channels using NNK graphs. Unlike standard methods, it requires no validation set, enables channel-specific stopping times, and achieves state-of-the-art test accuracy with reduced training time by leveraging task-specific performance metrics and efficient LOO computation frequencies.

ABSTRACT

State-of-the-art neural network architectures continue to scale in size and deliver impressive generalization results, although this comes at the expense of limited interpretability. In particular, a key challenge is to determine when to stop training the model, as this has a significant impact on generalization. Convolutional neural networks (ConvNets) comprise high-dimensional feature spaces formed by the aggregation of multiple channels, where analyzing intermediate data representations and the model's evolution can be challenging owing to the curse of dimensionality. We present channel-wise DeepNNK (CW-DeepNNK), a novel channel-wise generalization estimate based on non-negative kernel regression (NNK) graphs with which we perform local polytope interpolation on low-dimensional channels. This method leads to instance-based interpretability of both the learned data representations and the relationship between channels. Motivated by our observations, we use CW-DeepNNK to propose a novel early stopping criterion that (i) does not require a validation set, (ii) is based on a task performance metric, and (iii) allows stopping to be reached at different points for each channel. Our experiments demonstrate that our proposed method has advantages as compared to the standard criterion based on validation set performance.

Motivation & Objective

• To address the challenge of determining optimal training stopping time in deep learning without relying on a validation set.
• To enable instance-based interpretability of learned feature representations by analyzing intermediate activations at the channel level.
• To develop a channel-wise generalization estimate that captures the unique contribution and learning dynamics of each convolutional filter.
• To propose an early stopping criterion that adapts per-channel training duration based on local performance, improving efficiency and generalization.
• To maintain high test accuracy while eliminating the need to reserve labeled data for validation, especially beneficial in low-data regimes.

Proposed method

• The method constructs NNK graphs on low-dimensional subvectors (channels) of intermediate layer activations, enabling local polytope interpolation for generalization estimation.
• It uses leave-one-out (LOO) classification performance on NNK polytopes as a per-channel generalization metric.
• A channel-wise early stopping rule is applied when the LOO performance on a given channel ceases to improve, allowing different stopping times per channel.
• The approach employs range-normalized cosine and Gaussian kernels for robustness in high-dimensional feature spaces.
• LOO computation is performed at reduced frequencies (e.g., every T epochs) to reduce computational cost while preserving performance.
• The framework is compatible with any optimizer and does not require hyperparameter tuning or validation data.

Experimental results

Research questions

• RQ1 Can per-channel generalization be estimated effectively without a validation set using local polytope interpolation?
• RQ2 Does channel-wise early stopping based on LOO performance improve test accuracy and reduce training time compared to standard validation-based methods?
• RQ3 Can the proposed method maintain high performance in low-data regimes by utilizing all labeled data for training?
• RQ4 How does the frequency of LOO computation affect the efficiency and accuracy of the early stopping criterion?
• RQ5 Can the method detect channels with low predictive utility, enabling future pruning for model compression?

Key findings

• CW-DeepNNK with the cosine kernel achieves the best trade-off between test accuracy and training iterations, outperforming both validation-based early stopping and full-vector DeepNNK.
• The method maintains or exceeds the test accuracy of validation-based early stopping while training on all labeled data, eliminating the need to reserve data for validation.
• Reducing LOO computation frequency to every 10 epochs preserves test accuracy and stopping time while achieving computational efficiency comparable to validation-based methods.
• In small-data settings (1000 samples), CW-DeepNNK achieves higher test accuracy than the validation-based approach due to full data utilization.
• The approach detects overfitting earlier than standard methods, allowing earlier stopping without sacrificing generalization performance.
• The method enables channel-specific training termination, with some channels reaching optimal performance before others, indicating differential learning rates across filters.