[Paper Review] Contrastive Learning for Interactive Recommendation in Fashion

Karin Sevegnani, Arjun Seshadri+5 more|arXiv (Cornell University)|Jul 25, 2022

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes WhisperLite, a contrastive learning-based model that recommends personalized fashion items from user-provided text requests without relying on historical user data. By leveraging CLIP embeddings and a composite loss combining binary cross-entropy and contrastive loss, WhisperLite achieves state-of-the-art performance on a real-world fashion dataset and demonstrates superior relevance in human evaluations compared to random baselines.

ABSTRACT

Recommender systems and search are both indispensable in facilitating personalization and ease of browsing in online fashion platforms. However, the two tools often operate independently, failing to combine the strengths of recommender systems to accurately capture user tastes with search systems' ability to process user queries. We propose a novel remedy to this problem by automatically recommending personalized fashion items based on a user-provided text request. Our proposed model, WhisperLite, uses contrastive learning to capture user intent from natural language text and improves the recommendation quality of fashion products. WhisperLite combines the strength of CLIP embeddings with additional neural network layers for personalization, and is trained using a composite loss function based on binary cross entropy and contrastive loss. The model demonstrates a significant improvement in offline recommendation retrieval metrics when tested on a real-world dataset collected from an online retail fashion store, as well as widely used open-source datasets in different e-commerce domains, such as restaurants, movies and TV shows, clothing and shoe reviews. We additionally conduct a user study that captures user judgements on the relevance of the model's recommended items, confirming the relevancy of WhisperLite's recommendations in an online setting.

Motivation & Objective

• To address the gap between standalone recommender systems and search tools in fashion e-commerce by enabling personalized recommendations from free-form user text requests.
• To model user intent from natural language descriptions without relying on historical click-stream or purchase data, especially for cold-start users.
• To improve recommendation quality in the presence of noisy, ambiguous, or contradictory user requests and label noise in stylist-annotated data.
• To validate the model’s generalization across diverse domains using real-world and open-source datasets.
• To evaluate the model’s real-world performance through a human study on Amazon Mechanical Turk.

Proposed method

• WhisperLite combines pre-trained CLIP image and text encoders with fine-tuned feedforward networks to map user text requests and fashion product features into a shared embedding space.
• The model is trained using a composite loss function that combines binary cross-entropy for item relevance prediction and contrastive loss to align relevant text-product pairs.
• Contrastive learning is used to pull together positive pairs (relevant text-request and product) and push apart negative pairs (irrelevant combinations) in the embedding space.
• The method avoids fine-tuning the lower layers of the CLIP model, instead freezing them and only training the top two perceptrons to preserve generalization and reduce noise interference.
• The model is evaluated both offline on multiple datasets and online via a human study with qualified MTurk workers rating recommendation relevance on a 5-point scale.
• Ablation studies compare WhisperLite to a full fine-tuning baseline (Whisper), showing that freezing CLIP layers improves performance on noisy, real-world data.

Experimental results

Research questions

• RQ1 Can a contrastive learning approach effectively map free-form user text requests to relevant fashion product recommendations without historical user data?
• RQ2 How does the choice of fine-tuning strategy (full vs. partial) impact performance on noisy, real-world fashion recommendation data?
• RQ3 To what extent does the proposed model generalize across diverse e-commerce domains such as clothing, movies, restaurants, and shoes?
• RQ4 How do human users perceive the relevance of recommendations generated by WhisperLite compared to random baselines in an online setting?
• RQ5 Does the use of CLIP embeddings with a contrastive loss improve retrieval performance on complex, ambiguous user requests?

Key findings

• WhisperLite significantly outperforms the random baseline in human evaluation, achieving an average relevance rating of 0.52 (k=3), 0.72 (k=5), and 0.59 (k=7), all substantially higher than the random baseline’s scores around 0.31–0.38.
• The model’s performance is more consistent and reliable, as indicated by smaller error bars in human evaluation, especially at k=5 and k=7, where the standard deviation is lower than the mean for the random baseline.
• On the real-world WhisperD dataset, WhisperLite outperforms the full-fine-tuning baseline Whisper, suggesting that freezing CLIP’s lower layers prevents noise-induced degradation in feature learning.
• WhisperLite generalizes well across domains, achieving strong results on open-source datasets such as Yelp, Amazon clothing, and Amazon movies, with performance closer to the Amazon movies dataset than to the more challenging WhisperD set.
• The composite loss function combining binary cross-entropy and contrastive loss effectively improves retrieval metrics, particularly in handling ambiguous or incomplete user requests.
• The human study confirms that WhisperLite’s recommendations are perceived as more relevant and coherent by users, with a net preference for model outputs over random suggestions across all k-values.