[Paper Review] Generating Diverse and Informative Natural Language Fashion Feedback

Gil Sadeh, Lior Fritz+2 more|arXiv (Cornell University)|Jun 15, 2019

Multimodal Machine Learning Applications28 references4 citations

TL;DR

This paper proposes a maximum mutual information (MMI)-based decoding method for generating diverse and informative natural language fashion feedback on outfit images. By modifying standard image captioning models with visual attention and MMI decoding, the approach produces more specific, varied, and human-like responses than standard maximum likelihood training, achieving performance comparable to human fashion experts in human evaluations.

ABSTRACT

Recent advances in multi-modal vision and language tasks enable a new set of applications. In this paper, we consider the task of generating natural language fashion feedback on outfit images. We collect a unique dataset, which contains outfit images and corresponding positive and constructive fashion feedback. We treat each feedback type separately, and train deep generative encoder-decoder models with visual attention, similar to the standard image captioning pipeline. Following this approach, the generated sentences tend to be too general and non-informative. We propose an alternative decoding technique based on the Maximum Mutual Information objective function, which leads to more diverse and detailed responses. We evaluate our model with common language metrics, and also show human evaluation results. This technology is applied within the ``Alexa, how do I look?'' feature, publicly available in Echo Look devices.

Motivation & Objective

• To address the lack of diversity and specificity in standard image captioning models when generating fashion feedback.
• To develop a generative model that captures subtle, abstract fashion concepts like fit, trend, and color combination not directly visible in images.
• To improve response quality for the 'Alexa, how do I look?' feature in Echo Look devices by generating more informative and varied feedback.
• To overcome limitations of standard NLP metrics (e.g., BLEU, CIDEr) that fail to capture diversity in fashion feedback due to low n-gram overlap.
• To validate model performance through human evaluation, including a fashion Turing test and language quality assessment.

Proposed method

• Collect a unique dataset of ~60K outfit images with ~170K paired positive and constructive fashion feedback sentences.
• Train end-to-end encoder-decoder models using a visual attention mechanism, with a CNN as the image encoder and an RNN as the decoder.
• Apply a maximum mutual information (MMI) objective during decoding to reduce reliance on sentence priors and increase response diversity.
• Use a top-down attention mechanism to focus on salient image regions and improve relevance of generated feedback.
• Incorporate a temperature decay strategy with β=0.4, reducing MMI influence after 11 and 16 steps for the GOOD and TIP models, respectively, to reduce language errors.
• Evaluate using both automatic metrics (especially CIDEr) and extensive human evaluation, including a fashion Turing test and language quality assessment.

Experimental results

Research questions

• RQ1 Can a deep generative model with visual attention produce diverse and informative fashion feedback that matches human-level quality?
• RQ2 Does MMI-based decoding significantly improve the diversity and specificity of fashion feedback compared to standard maximum likelihood training?
• RQ3 To what extent can generated fashion feedback fool human evaluators in a fashion Turing test?
• RQ4 How do standard NLP metrics like CIDEr perform in evaluating fashion feedback, given the low n-gram overlap in human-annotated responses?
• RQ5 Can the model generate feedback that reflects abstract fashion concepts not directly visually grounded in the image?

Key findings

• The proposed model with top-down attention and MMI decoding achieved a CIDEr score of 0.737 on the TIP task and 0.988 on the GOOD task, outperforming both baseline models and even the Fashion Specialists' performance on automatic metrics.
• Human evaluation showed that 83% of generated GOOD feedback and 84% of TIP feedback were classified as human-written in the fashion Turing test, indicating high realism and relevance.
• In the language quality test, 89% of generated GOOD responses and 93% of TIP responses were judged free of language errors, comparable to the 86% and 88% success rate of ground truth human responses.
• The model significantly improved diversity and vocabulary usage compared to standard maximum likelihood decoding, as evidenced by higher CIDEr and vocabulary diversity scores.
• Despite strong automatic metric performance, the gap in diversity and vocabulary usage suggests that current metrics still do not fully capture the nuances of high-quality fashion feedback.
• The model is successfully deployed in the 'Alexa, how do I look?' feature on Echo Look devices, demonstrating real-world applicability and user experience improvement.