[論文レビュー] Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer

David Berthelot, Colin Raffel+2 more|arXiv (Cornell University)|Jul 19, 2018

Model Reduction and Neural Networks被引用数 63

ひとこと要約

本論文は ACAI を導入し、敵対的正則化子がオートエンコーダの補間の現実性を向上させ、下流の表現学習を強化することを示し、合成ベンチマークと複数のデータセットで検証されている。

ABSTRACT

Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code. In some cases, autoencoders can "interpolate": By decoding the convex combination of the latent codes for two datapoints, the autoencoder can produce an output which semantically mixes characteristics from the datapoints. In this paper, we propose a regularization procedure which encourages interpolated outputs to appear more realistic by fooling a critic network which has been trained to recover the mixing coefficient from interpolated data. We then develop a simple benchmark task where we can quantitatively measure the extent to which various autoencoders can interpolate and show that our regularizer dramatically improves interpolation in this setting. We also demonstrate empirically that our regularizer produces latent codes which are more effective on downstream tasks, suggesting a possible link between interpolation abilities and learning useful representations.

研究の動機と目的

• Formalize and evaluate interpolation in autoencoders.
• Propose an adversarial regularizer to improve interpolations.
• Create a quantitative benchmark to measure interpolation quality.
• Demonstrate that better interpolation correlates with improved representations for downstream tasks.

提案手法

• Define interpolation via convex combinations in latent space and decode with a neural autoencoder.
• Introduce a critic that predicts the interpolation coefficient alpha from interpolated reconstructions.
• Train the autoencoder to fool the critic, using a regularizer that pushes interpolated points to look like reconstructions of real data.
• Develop a synthetic 32x32 line-drawing benchmark with controllable interpolation along a known manifold Lambda to quantify interpolation quality.
• Compare ACAI to standard autoencoders, VAEs, AAEs, and VQ-VAE on interpolation metrics and downstream tasks.

実験結果

リサーチクエスチョン

• RQ1 How can interpolation quality in autoencoders be quantified on a well-defined manifold?
• RQ2 Does adversarial regularization (ACAI) produce more realistic and smooth interpolations than standard autoencoders?
• RQ3 How does improving interpolation affect latent representations for downstream tasks like classification and clustering?
• RQ4 What is the performance of ACAI relative to other autoencoder variants across datasets and latent dimensions?

主な発見

Model	Mean Distance (×10^-3)	Smoothness
Baseline	6.88 ± 0.21	0.44 ± 0.04
Dropout	2.85 ± 0.54	0.74 ± 0.02
Denoising	4.21 ± 0.32	0.66 ± 0.02
VAE	1.21 ± 0.17	0.49 ± 0.13
AAE	3.26 ± 0.19	0.14 ± 0.02
VQ-VAE	5.41 ± 0.49	0.77 ± 0.02
ACAI	0.24 ± 0.01	0.10 ± 0.01

• ACAI achieves the best Mean Distance and Smoothness scores on the synthetic lines benchmark among the models tested.
• ACAI substantially improves downstream representation learning, with notable gains on MNIST, SVHN, and CIFAR-10 for single-layer classifiers.
• On SVHN with dz=256, ACAI reaches 85.14% accuracy, far above the baseline (22.74%).
• ACAI generally yields realistic interpolations and smooth morphological transitions, outperforming VAEs, denoising autoencoders, and VQ-VAE in interpolation quality.
• Latent representations from ACAI enhance clustering performance on MNIST and SVHN compared to other autoencoders.