[Paper Review] Exploiting Latent Codes: Interactive Fashion Product Generation, Similar Image Retrieval, and Cross-Category Recommendation using Variational Autoencoders

James-Andrew Sarmiento|arXiv (Cornell University)|Sep 2, 2020

Generative Adversarial Networks and Image Synthesis14 references4 citations

TL;DR

This paper proposes a unified variational autoencoder (VAE)-based framework for interactive fashion product generation, similar image retrieval, and cross-category recommendation. By exploiting learned latent codes, the method enables user-controlled product design via sliders, uses log-likelihood and Euclidean distance on fixed-epsilon sampling for retrieval, and achieves high performance with mAP up to 0.95 and F1 scores of 0.98 on clustering tasks.

ABSTRACT

The rise of deep learning applications in the fashion industry has fueled advances in curating large-scale datasets to build applications for product design, image retrieval, and recommender systems. In this paper, the author proposes using Variational Autoencoder (VAE) to build an interactive fashion product application framework that allows the users to generate products with attributes according to their liking, retrieve similar styles for the same product category, and receive content-based recommendations from other categories. Fashion product images dataset containing eyewear, footwear, and bags are appropriate to illustrate that this pipeline is applicable in the booming industry of e-commerce enabling direct user interaction in specifying desired products paired with new methods for data matching, and recommendation systems by using VAE and exploiting its generated latent codes.

Motivation & Objective

• Address the need for interactive, user-driven fashion product design in e-commerce by enabling customization through attribute sliders.
• Improve similarity retrieval within fashion categories by leveraging disentangled and regularized latent representations from VAEs.
• Enable cross-category fashion recommendations by matching user-generated product concepts to items in other categories using latent space clustering.
• Demonstrate the feasibility of using a single VAE architecture to solve multiple e-commerce tasks: generation, retrieval, and recommendation.
• Evaluate the effectiveness of two latent code utilization strategies—log-likelihood maximization and fixed-epsilon sampling—for image retrieval and clustering.

Proposed method

• Train a VAE on a fashion dataset (eyewear, footwear, bags) to learn a regularized, continuous latent space with mean μ and standard deviation σ.
• Apply the reparameterization trick (z = μ + σ² ⊙ ε, ε ~ N(0,1)) to enable backpropagation and end-to-end training of the VAE.
• Use the learned latent codes for interactive product generation by interpolating or modifying latent vectors via user-controlled sliders.
• Perform similar image retrieval using two methods: (1) maximizing log-likelihood of latent codes and (2) computing Euclidean distance on fixed-epsilon sampled latent vectors.
• Cluster fashion items using K-means on latent codes, with cluster centers used to guide cross-category recommendations.
• Generate cross-category recommendations by finding the most similar image in a different category’s cluster using the same retrieval techniques.

Experimental results

Research questions

• RQ1 Can a single VAE architecture effectively support interactive fashion product generation, similar image retrieval, and cross-category recommendation?
• RQ2 How do different strategies for utilizing latent codes—log-likelihood maximization versus Euclidean distance on fixed-epsilon sampling—affect retrieval performance?
• RQ3 To what extent can VAE-learned latent codes support meaningful clustering and cross-category recommendations in fashion e-commerce?
• RQ4 How does the framework perform in terms of retrieval accuracy and clustering quality under imbalanced data conditions?
• RQ5 Can user-controlled attribute adjustments in the latent space lead to semantically coherent and visually plausible fashion product generations?

Key findings

• The framework achieved a mean average precision (mAP) of up to 0.95 for top-10, top-25, and top-50 similar image retrieval, with fixed-epsilon sampling outperforming log-likelihood maximization.
• K-means clustering using latent codes achieved an F1 score as high as 0.98, even with imbalanced data, indicating strong clustering quality.
• The fixed-epsilon sampling method produced more coherent cross-category recommendations, such as pairing bluish-tinted eyewear with blue bags and shoes, based on color and pattern consistency.
• Log-likelihood maximization prioritized texture and color similarity, while fixed-epsilon sampling emphasized structural and attribute-level consistency in recommendations.
• Both retrieval methods retrieved the same image as the top match, but diverged in second- and third-best matches, demonstrating distinct retrieval behaviors.
• The unified VAE pipeline successfully integrated product generation, retrieval, and recommendation into a single architecture, reducing the need for multiple models in e-commerce systems.