[Paper Review] Neural Networks for Fashion Image Classification and Visual Search

Fengzi Li, Shashi Kant+3 more|arXiv (Cornell University)|May 17, 2020

Advanced Image and Video Retrieval Techniques2 references4 citations

TL;DR

This paper proposes a dual deep learning approach for fashion e-commerce: (1) transfer learning with VGG19 and Inception V3 for accurate image classification of fashion products, and (2) autoencoder-based visual search using cosine similarity to retrieve similar items from images. The key contribution is a practical pipeline enabling automated tagging for sellers and visual search for buyers, achieving high accuracy on a real-world fashion dataset with limited resolution and class imbalance.

ABSTRACT

We discuss two potentially challenging problems faced by the ecommerce industry. One relates to the problem faced by sellers while uploading pictures of products on the platform for sale and the consequent manual tagging involved. It gives rise to misclassifications leading to its absence from search results. The other problem concerns with the potential bottleneck in placing orders when a customer may not know the right keywords but has a visual impression of an image. An image based search algorithm can unleash the true potential of ecommerce by enabling customers to click a picture of an object and search for similar products without the need for typing. In this paper, we explore machine learning algorithms which can help us solve both these problems.

Motivation & Objective

• To address the challenge of manual product tagging by sellers, which often leads to misclassification and poor search visibility.
• To solve the buyer-side problem of keyword dependency in product search by enabling image-based visual search.
• To develop a scalable, accurate image classification system using pre-trained CNNs on a real-world fashion dataset.
• To implement and evaluate a visual search system using autoencoders and cosine similarity for similar product retrieval.
• To assess model performance under data imbalance and low-resolution image constraints common in e-commerce.

Proposed method

• Fine-tune pre-trained VGG19 and Inception V3 models on the Fashion Product Images (Small) dataset for image classification.
• Use transfer learning to extract features from convolutional layers and train fully connected layers for classification.
• Train deep autoencoders using encoder-decoder architecture to learn low-dimensional latent representations of fashion images.
• Apply cosine similarity to compare image embeddings and retrieve top-k most similar images in the visual search pipeline.
• Implement data filtration to reduce minority classes and apply class re-weighting to mitigate data imbalance.
• Utilize Google Colab with GPU/TPU for efficient training and leverage TensorFlow for model implementation.

Experimental results

Research questions

• RQ1 Can pre-trained CNNs like VGG19 and Inception V3 achieve high accuracy in classifying fashion product images despite class imbalance?
• RQ2 How effective are autoencoders in learning meaningful image embeddings for visual search in fashion e-commerce?
• RQ3 To what extent does cosine similarity between learned embeddings retrieve semantically and visually similar fashion products?
• RQ4 How do low-resolution images and data imbalance affect the performance of image classification and visual search models?
• RQ5 Can visual search using external images (e.g., web-sourced photos) generalize to real-world user queries?

Key findings

• The Inception V3 model achieved the highest accuracy among tested architectures, with a reported performance of 93.7% on the image classification task.
• Autoencoder-based visual search successfully retrieved semantically and visually similar fashion items, with qualitative results showing high relevance for common categories like t-shirts and earrings.
• The model demonstrated generalization capability by retrieving relevant products from the dataset when using external images as queries.
• Despite low-resolution images (80×60 pixels), the models achieved strong performance, indicating feasibility for real-time deployment.
• Class imbalance remained a challenge, particularly for rare categories like wristbands (only 7 images), which limited model generalization for such classes.
• The use of data filtration and class re-weighting improved model robustness, though minority class misclassification persisted.