[Paper Review] Large Multi-modal Encoders for Recommendation

Zixuan Yi, Zijun Long+3 more|arXiv (Cornell University)|Oct 31, 2023

Recommender Systems and Techniques4 citations

TL;DR

This paper investigates the integration of large multi-modal (LMM) encoders—CLIP and VLMo—into multi-modal recommendation systems, demonstrating that pre-trained and fine-tuned LMM encoders significantly improve user and item representation learning by enabling deeper cross-modal alignment. The study shows that end-to-end training further enhances performance, especially with dual-stream encoders like CLIP, leading to state-of-the-art results across three multi-modal recommendation datasets.

ABSTRACT

In recent years, the rapid growth of online multimedia services, such as e-commerce platforms, has necessitated the development of personalised recommendation approaches that can encode diverse content about each item. Indeed, modern multi-modal recommender systems exploit diverse features obtained from raw images and item descriptions to enhance the recommendation performance. However, the existing multi-modal recommenders primarily depend on the features extracted individually from different media through pre-trained modality-specific encoders, and exhibit only shallow alignments between different modalities - limiting these systems' ability to capture the underlying relationships between the modalities. In this paper, we investigate the usage of large multi-modal encoders within the specific context of recommender systems, as these have previously demonstrated state-of-the-art effectiveness when ranking items across various domains. Specifically, we tailor two state-of-the-art multi-modal encoders (CLIP and VLMo) for recommendation tasks using a range of strategies, including the exploration of pre-trained and fine-tuned encoders, as well as the assessment of the end-to-end training of these encoders. We demonstrate that pre-trained large multi-modal encoders can generate more aligned and effective user/item representations compared to existing modality-specific encoders across three multi-modal recommendation datasets. Furthermore, we show that fine-tuning these large multi-modal encoders with recommendation datasets leads to an enhanced recommendation performance. In terms of different training paradigms, our experiments highlight the essential role of the end-to-end training of large multi-modal encoders in multi-modal recommendation systems.

Motivation & Objective

• To address the limitation of shallow cross-modal alignment in existing multi-modal recommender systems that rely on modality-specific encoders.
• To investigate whether large multi-modal encoders (LMMs) such as CLIP and VLMo can improve representation learning and recommendation performance in multi-modal settings.
• To compare the effectiveness of different training paradigms—pre-training, fine-tuning, and end-to-end training—on LMM encoders within recommendation models.
• To analyze the contribution of individual modalities (visual and textual) when enhanced by LMM encoders in multi-modal recommendation.
• To identify architectural and training-related conflicts between LMM encoders and existing recommendation models, such as LATTICE.

Proposed method

• The study integrates two state-of-the-art LMM encoders—CLIP (dual-stream) and VLMo (unified)—into five existing multi-modal recommendation models: MMGCL, LATTICE, and three others.
• Pre-trained CLIP and VLMo encoders are used as feature extractors, with fine-tuning performed on item image and text data from each dataset to adapt the encoders to recommendation tasks.
• The paper evaluates three training paradigms: (1) using pre-trained encoders only, (2) fine-tuning the encoders on recommendation data, and (3) end-to-end training of the entire model with the LMM encoder.
• Cross-modal alignment is enhanced through contrastive learning objectives inherent in CLIP and VLMo, which jointly embed visual and textual features into a shared space.
• Performance is evaluated using standard metrics such as NDCG@20 and Recall@K across three multi-modal recommendation datasets: Amazon Sports, Amazon Clothing, and Amazon Baby.
• A modality contribution analysis is conducted to compare performance using only visual, only textual, or both modalities as input, with and without LMM encoders.

Experimental results

Research questions

• RQ1 RQ1: Can pre-trained large multi-modal encoders (CLIP and VLMo) improve the performance of multi-modal recommendation models compared to modality-specific encoders?
• RQ2 RQ2: How does fine-tuning the LMM encoders on recommendation datasets affect recommendation performance?
• RQ3 RQ3: What is the impact of different training paradigms—two-step (pre-train + fine-tune) versus end-to-end training—on the effectiveness of LMM encoders in multi-modal recommendation?
• RQ4 RQ4: To what extent do LMM encoders enhance the contribution of individual modalities (visual and textual) in multi-modal recommendation?
• RQ5 RQ5: Are there architectural conflicts between LMM encoders and certain recommendation models (e.g., LATTICE), and how do they affect performance?

Key findings

• Pre-trained CLIP and VLMo encoders improved recommendation performance in 79% of 120 tested cases compared to modality-specific encoders, demonstrating significant gains in cross-modal alignment and representation quality.
• Fine-tuning the LMM encoders on recommendation datasets led to further performance improvements, confirming the benefit of domain-specific adaptation.
• End-to-end training of CLIP (a dual-stream encoder) significantly enhanced performance across all models, particularly resolving conceptual conflicts with models like LATTICE.
• In contrast, VLMo (a unified encoder) did not show the same performance gains from end-to-end training, suggesting architectural differences affect training dynamics.
• The integration of LMM encoders consistently improved performance when using both visual and textual modalities, with statistically significant gains (p < 0.05) observed in NDCG@20 scores across all datasets.
• Modality contribution analysis confirmed that LMM encoders enable deeper alignment, making multi-modal fusion more effective than single-modality inputs, even when baseline models underperformed with fused features.