[Paper Review] FashionViL: Fashion-Focused Vision-and-Language Representation Learning

Han Xiao, Licheng Yu+4 more|arXiv (Cornell University)|Jul 17, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

FashionViL proposes a novel vision-and-language representation learning framework tailored for the fashion domain, introducing multi-view contrastive learning and pseudo-attributes classification to exploit multi-image inputs and fine-grained textual attributes. It achieves state-of-the-art performance across five downstream fashion tasks, demonstrating superior generalization and flexibility via a modality-agnostic Transformer architecture.

ABSTRACT

Large-scale Vision-and-Language (V+L) pre-training for representation learning has proven to be effective in boosting various downstream V+L tasks. However, when it comes to the fashion domain, existing V+L methods are inadequate as they overlook the unique characteristics of both the fashion V+L data and downstream tasks. In this work, we propose a novel fashion-focused V+L representation learning framework, dubbed as FashionViL. It contains two novel fashion-specific pre-training tasks designed particularly to exploit two intrinsic attributes with fashion V+L data. First, in contrast to other domains where a V+L data point contains only a single image-text pair, there could be multiple images in the fashion domain. We thus propose a Multi-View Contrastive Learning task for pulling closer the visual representation of one image to the compositional multimodal representation of another image+text. Second, fashion text (e.g., product description) often contains rich fine-grained concepts (attributes/noun phrases). To exploit this, a Pseudo-Attributes Classification task is introduced to encourage the learned unimodal (visual/textual) representations of the same concept to be adjacent. Further, fashion V+L tasks uniquely include ones that do not conform to the common one-stream or two-stream architectures (e.g., text-guided image retrieval). We thus propose a flexible, versatile V+L model architecture consisting of a modality-agnostic Transformer so that it can be flexibly adapted to any downstream tasks. Extensive experiments show that our FashionViL achieves a new state of the art across five downstream tasks. Code is available at https://github.com/BrandonHanx/mmf.

Motivation & Objective

• To address the limitations of generic vision-and-language pre-training models in the fashion domain, which fail to exploit unique characteristics of fashion data such as multi-view images and fine-grained attributes.
• To design a flexible, versatile model architecture that can adapt to diverse fashion downstream tasks, including those not suited for standard one-stream or two-stream designs.
• To improve multimodal representation learning by introducing two novel pre-training tasks specifically tailored to fashion data's intrinsic properties.
• To achieve state-of-the-art performance across five diverse fashion-focused vision-and-language downstream tasks.

Proposed method

• Proposes a Multi-View Contrastive Learning task that aligns the visual representation of one image with the compositional multimodal representation of another image-text pair, leveraging multiple views of the same fashion item.
• Introduces a Pseudo-Attributes Classification task to encourage unimodal representations (visual and textual) of the same fine-grained attribute to be close in the embedding space, enhancing attribute-level alignment.
• Employs a modality-agnostic Transformer architecture that enables flexible adaptation to various downstream tasks, including text-guided image retrieval, which does not fit standard one- or two-stream paradigms.
• Utilizes a VQ-VAE-based image tokenization method to learn discrete visual representations, improving training efficiency and representation quality.
• Applies contrastive and cross-entropy losses during fine-tuning, with task-specific hyper-parameters and data augmentation strategies for each downstream task.
• Reimplements and releases new data splits for the OCIR benchmark to ensure reproducibility, as original code and splits were unavailable.

Experimental results

Research questions

• RQ1 Can multi-view contrastive learning improve representation learning in fashion V+L by leveraging multiple images of the same item?
• RQ2 Does pseudo-attribute classification enhance the model's ability to capture fine-grained visual and textual attributes in fashion descriptions?
• RQ3 Can a modality-agnostic Transformer architecture effectively handle diverse fashion V+L downstream tasks, including non-standard retrieval setups?
• RQ4 To what extent does pre-training on fashion-specific objectives improve performance on downstream tasks compared to generic V+L models?

Key findings

• FashionViL achieves state-of-the-art performance on five downstream fashion tasks, including text-guided image retrieval, outfit complementary item retrieval, and category/subcategory recognition.
• The model achieves 84.30% accuracy on multi-image subcategory recognition (M-SCR) with pre-training, compared to 83.02% without, demonstrating the effectiveness of the proposed pre-training tasks.
• The pre-trained model significantly improves performance on category recognition (CR) and subcategory recognition (SCR), with gains of 13.32% and 11.14% in macro-F1, respectively, over non-pretrained baselines.
• t-SNE visualizations show that pre-training leads to better clustering of multimodal representations at both category and subcategory levels, confirming improved semantic alignment.
• The model outperforms existing SOTA methods on the OCIR benchmark, with a significant gain over the original CSA-Net implementation, despite using a re-implemented data split.
• Ablation studies confirm that both the Multi-View Contrastive Learning and Pseudo-Attributes Classification tasks contribute significantly to the final performance, with the latter showing strong gains on fine-grained attribute understanding.