[Paper Review] Ranking Deep Learning Generalization using Label Variation in Latent Geometry Graphs

Carlos Lassance, Louis Béthune+3 more|arXiv (Cornell University)|Nov 25, 2020

Advanced Graph Neural Networks12 references4 citations

TL;DR

This paper proposes a generalization measure for deep neural networks using Latent Geometry Graphs (LGGs) and label variation, achieving 3rd place in the NeurIPS 2020 PGDL competition. By constructing similarity-based graphs from intermediate layer representations and measuring label variation—i.e., the sum of edge weights between samples of different classes—it quantifies how well the latent space aligns with classification tasks, yielding a robust proxy for generalization without requiring a validation set.

ABSTRACT

Measuring the generalization performance of a Deep Neural Network (DNN) without relying on a validation set is a difficult task. In this work, we propose exploiting Latent Geometry Graphs (LGGs) to represent the latent spaces of trained DNN architectures. Such graphs are obtained by connecting samples that yield similar latent representations at a given layer of the considered DNN. We then obtain a generalization score by looking at how strongly connected are samples of distinct classes in LGGs. This score allowed us to rank 3rd on the NeurIPS 2020 Predicting Generalization in Deep Learning (PGDL) competition.

Motivation & Objective

• To develop a validation-set-free proxy for deep learning generalization performance.
• To address the limitation of traditional validation-based generalization estimation, which reduces training data and may degrade model performance.
• To explore whether latent space geometry, captured via LGGs, can serve as a reliable indicator of generalization capacity.
• To improve generalization prediction robustness across diverse architectures and hyperparameters by incorporating mixup-augmented data.
• To identify a generalization score that correlates strongly with actual test performance while minimizing overfitting artifacts.

Proposed method

• Construct Latent Geometry Graphs (LGGs) by connecting data samples based on similarity of their intermediate layer representations using cosine or RBF similarity kernels.
• Apply k-nearest neighbor thresholding to create sparse, locally connected graphs, optionally symmetrizing and normalizing using degree matrices.
• Define label variation as the trace of the product of the label signal and the graph Laplacian, measuring the total edge weight between samples of different classes.
• Use mixup-augmented training samples to generate more robust LGGs, reducing overfitting bias in the generalization score.
• Compute the median label variation across multiple LGGs (|G|) to improve stability and reduce variance in the final score.
• Introduce the Variation Per Class Median (VPM) score as the primary generalization proxy, combining LGGs, label variation, and mixup augmentation.

Experimental results

Research questions

• RQ1 Can label variation in Latent Geometry Graphs serve as a reliable, validation-set-free proxy for deep learning generalization?
• RQ2 How does incorporating mixup-augmented data improve the robustness and generalization of the label variation score across different architectures?
• RQ3 Does the median of multiple LGG-based label variation scores reduce variance and improve consistency between public and development sets?
• RQ4 How do different LGG construction strategies (e.g., similarity kernel, k-NN thresholding) affect the correlation with actual generalization performance?
• RQ5 To what extent does the proposed method generalize across diverse network architectures and training hyperparameters without requiring a validation set?

Key findings

• The proposed VPM (Variation Per Class Median) score achieved a final mean score of 9.99 on the NeurIPS 2020 PGDL competition final set, ranking 3rd out of 24 submissions.
• The VPM score demonstrated better balance between public and development set performance compared to baseline scores like VR (Variation Rate) and WCV (Weighted Class Variation), which overfitted to the public set.
• Using mixup-augmented samples significantly improved generalization of the label variation metric, reducing overfitting artifacts in highly overfitted models.
• The final submission used only one LGG (|G|=1) due to time constraints, resulting in a performance drop on the public set (from 11.22 to 6.26), indicating that increasing graph count improves accuracy.
• Task-specific results showed high variability (e.g., 16.23 on Task 8, 2.28 on Task 9), suggesting that the method's performance is sensitive to data distribution and that increasing graph diversity could mitigate this variance.
• The method achieved strong consistency across tasks, with the final solution running in under 10% of the time budget on most tasks, indicating high computational efficiency and scalability.