[Paper Review] Fashion-IQ 2020 Challenge 2nd Place Team's Solution

Minchul Shin, Yoon‐Jae Cho+1 more|arXiv (Cornell University)|Jul 13, 2020

Generative Adversarial Networks and Image Synthesis13 references4 citations

TL;DR

This paper presents RTIC, a novel multimodal composition method that learns the residual between target and candidate image features conditioned on text, using channel-wise attention modules across N blocks. The approach combines image and text encoders (ResNet-101/152, LSTM/GRU with GloVe embeddings) and achieves 2nd place in the Fashion-IQ 2020 challenge with a test score of 48.02 via an iterative ensemble strategy using Bayesian optimization to weight similarity scores from diverse models.

ABSTRACT

This paper is dedicated to team VAA's approach submitted to the Fashion-IQ challenge in CVPR 2020. Given a pair of the image and the text, we present a novel multimodal composition method, RTIC, that can effectively combine the text and the image modalities into a semantic space. We extract the image and the text features that are encoded by the CNNs and the sequential models (e.g., LSTM or GRU), respectively. To emphasize the meaning of the residual of the feature between the target and candidate, the RTIC is composed of N-blocks with channel-wise attention modules. Then, we add the encoded residual to the feature of the candidate image to obtain a synthesized feature. We also explored an ensemble strategy with variants of models and achieved a significant boost in performance comparing to the best single model. Finally, our approach achieved 2nd place in the Fashion-IQ 2020 Challenge with a test score of 48.02 on the leaderboard.

Motivation & Objective

• To improve fashion image retrieval by effectively composing image and text modalities into a shared semantic space.
• To model the residual difference between target and candidate image features using text-conditioned attention mechanisms.
• To enhance retrieval performance through an iterative ensemble strategy that combines multiple distinct models with optimized weighting.
• To achieve state-of-the-art performance on the Fashion-IQ 2020 benchmark without pretraining on external datasets.

Proposed method

• Proposes RTIC, a multimodal composer that models the residual between target and candidate image features using N blocks with channel-wise attention for gating.
• Uses a two-branch encoder: ResNet-101/152 for images and a 2-layered LSTM+GRU with 900D GloVe-initialized word embeddings for text.
• Applies triplet loss with hard negative mining and AdamW optimizer, with fine-tuned learning rates for image and text encoders.
• Employs an iterative ensemble process where the best validation score matrix is used as a new candidate, with optimal weights found via Bayesian optimization (hyperopt).
• Integrates knowledge distillation by aligning image encoder and multimodal composer outputs with off-the-shelf IR features from DeepFashion.
• Uses data augmentation including random horizontal flip, affine transformation, and image resizing to 224×224 with ratio-preserving padding.

Experimental results

Research questions

• RQ1 Can a residual-based multimodal composition method outperform existing methods like TIRG in fashion image retrieval?
• RQ2 How effective is an iterative ensemble strategy that progressively incorporates the best-performing score matrix into model combination?
• RQ3 To what extent can performance be improved by combining diverse models with optimized weighting, even without pretraining?
• RQ4 Does using a hybrid text encoder (LSTM + GRU) and higher-dimensional feature space improve retrieval accuracy?

Key findings

• The best single model using RTIC achieved 38.22 average recall on the validation set, a significant improvement over the baseline RTIC (33.24).
• The ensemble of four distinct models (Text-only, IR-match, TIRG, RTIC) achieved 45.05 on validation, showing a strong boost from model diversity.
• The iterative ensemble process improved the score from 45.05 to 47.55 on validation by progressively incorporating the best score matrix.
• Final test score reached 48.02 after including 80% of validation pairs in training and merging results from additional models, representing a 25.6% performance gain over the best single model.
• The use of Bayesian optimization for weight selection in the ensemble process was critical to maximizing test performance.
• No performance gain was observed from using alternative losses like SoftTriple, Multi-Similarity, or Circle loss, despite hyperparameter tuning.