[Paper Review] SimEx: Express Prediction of Inter-dataset Similarity by a Fleet of Autoencoders

Inseok Hwang, Jinho Lee+2 more|arXiv (Cornell University)|Jan 14, 2020

Domain Adaptation and Few-Shot Learning30 references4 citations

TL;DR

SimEx proposes a fast, training-free method for predicting inter-dataset similarity by evaluating reconstruction errors of unknown data through a fleet of pretrained autoencoders, each specialized for a distinct known data subset. It achieves over 10x speedup compared to transfer-learning-based baselines while maintaining strong correlation with performance-based similarity metrics.

ABSTRACT

Knowing the similarity between sets of data has a number of positive implications in training an effective model, such as assisting an informed selection out of known datasets favorable to model transfer or data augmentation problems with an unknown dataset. Common practices to estimate the similarity between data include comparing in the original sample space, comparing in the embedding space from a model performing a certain task, or fine-tuning a pretrained model with different datasets and evaluating the performance changes therefrom. However, these practices would suffer from shallow comparisons, task-specific biases, or extensive time and computations required to perform comparisons. We present SimEx, a new method for early prediction of inter-dataset similarity using a set of pretrained autoencoders each of which is dedicated to reconstructing a specific part of known data. Specifically, our method takes unknown data samples as input to those pretrained autoencoders, and evaluate the difference between the reconstructed output samples against their original input samples. Our intuition is that, the more similarity exists between the unknown data samples and the part of known data that an autoencoder was trained with, the better chances there could be that this autoencoder makes use of its trained knowledge, reconstructing output samples closer to the originals. We demonstrate that our method achieves more than 10x speed-up in predicting inter-dataset similarity compared to common similarity-estimating practices. We also demonstrate that the inter-dataset similarity estimated by our method is well-correlated with common practices and outperforms the baselines approaches of comparing at sample- or embedding-spaces, without newly training anything at the comparison time.

Motivation & Objective

• To address the challenge of efficiently estimating similarity between an unknown dataset and known reference datasets without retraining models.
• To reduce the computational cost of similarity estimation in transfer learning and data augmentation workflows.
• To develop a method that leverages pretrained autoencoders to infer similarity based on reconstruction quality, avoiding task-specific biases.
• To evaluate whether reconstruction error in the input space can serve as a reliable proxy for inter-dataset similarity.
• To demonstrate that SimEx provides a scalable, low-latency alternative to existing similarity estimation techniques in real-world ML pipelines.

Proposed method

• Train a dedicated autoencoder for each distinct data subset in known datasets, using either MSE or iSSIM as the reconstruction loss.
• Use the pretrained autoencoders to reconstruct unknown data samples, measuring the difference between input and reconstructed outputs.
• Compute similarity between unknown data and each known data subset based on reconstruction error (e.g., MSE or iSSIM between input and output).
• Use the reconstruction error as a proxy for similarity: lower error indicates higher similarity to the corresponding known data subset.
• Avoid any fine-tuning or additional training at inference time, enabling fast, real-time similarity prediction.
• Evaluate the method across multiple tasks, including transfer learning, data augmentation, and inter-class confusion prediction.

Experimental results

Research questions

• RQ1 Can reconstruction error from pretrained autoencoders serve as a reliable proxy for inter-dataset similarity?
• RQ2 How does SimEx compare to sample-space and embedding-space baselines in predicting similarity relevant to transfer learning and data augmentation?
• RQ3 To what extent does SimEx reduce runtime cost compared to transfer-learning-based similarity estimation?
• RQ4 How well does SimEx predict inter-class confusion levels in a dataset compared to baseline methods?
• RQ5 Does SimEx’s use of reconstruction error in the input space mitigate biases from model-specific feature representations?

Key findings

• SimEx achieves over 10x speedup in inter-dataset similarity prediction compared to transfer-learning-based baselines, as it requires no training at inference time.
• The similarity predicted by SimEx-MSE and SimEx-iSSIM shows a Spearman’s rho of 0.692 and 0.765, respectively, with confusion levels from 9-class LeNet-5 classifiers, indicating strong correlation.
• SimEx-iSSIM achieves a Spearman’s rho of 0.804 with confusion levels from 9-class ResNet-18 classifiers, outperforming baseline sample-iSSIM (rho = 0.338).
• SimEx’s similarity predictions are more consistent with model-based confusion patterns than sample-space distances, suggesting better predictive power for real-world ML tasks.
• The method maintains high correlation with performance-based similarity metrics while avoiding the computational burden of fine-tuning or feature extraction.
• SimEx outperforms both sample-space and embedding-space baselines in predicting inter-class confusion, indicating its potential for early dataset compatibility screening.