[Paper Review] Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecture

Jingyu Xu, Yang Wang|arXiv (Cornell University)|Dec 16, 2024

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes a multimodal Mixture-of-Experts (MOE) architecture enhanced with large language models (LLMs) to improve personalized healthcare recommendation systems. By integrating patient descriptions with limited image data, the model achieves superior performance on precision, recall, NDCG, and MAP@5 compared to standalone MOE or LLM baselines, though image inputs provided limited gains and were sensitive to image quality and reclassification issues.

ABSTRACT

With the increasing availability of multimodal data, many fields urgently require advanced architectures capable of effectively integrating these diverse data sources to address specific problems. This study proposes a hybrid recommendation model that combines the Mixture of Experts (MOE) framework with large language models to enhance the performance of recommendation systems in the healthcare domain. We built a small dataset for recommending healthy food based on patient descriptions and evaluated the model's performance on several key metrics, including Precision, Recall, NDCG, and MAP@5. The experimental results show that the hybrid model outperforms the baseline models, which use MOE or large language models individually, in terms of both accuracy and personalized recommendation effectiveness. The paper finds image data provided relatively limited improvement in the performance of the personalized recommendation system, particularly in addressing the cold start problem. Then, the issue of reclassification of images also affected the recommendation results, especially when dealing with low-quality images or changes in the appearance of items, leading to suboptimal performance. The findings provide valuable insights into the development of powerful, scalable, and high-performance recommendation systems, advancing the application of personalized recommendation technologies in real-world domains such as healthcare.

Motivation & Objective

• To address the challenge of integrating diverse multimodal healthcare data—especially text and images—into personalized recommendation systems.
• To improve recommendation accuracy and personalization in healthcare by combining the strengths of Mixture-of-Experts (MOE) and large language models (LLMs).
• To evaluate the impact of image data on recommendation performance, particularly in cold start scenarios and with low-quality or variable images.
• To develop a scalable, high-performance recommendation framework applicable to real-world healthcare domains such as personalized nutrition.

Proposed method

• The proposed architecture employs a Mixture-of-Experts (MOE) framework where multiple expert networks are dynamically activated based on input modality and context.
• Large language models (LLMs) are integrated to process and understand patient-provided textual descriptions, extracting semantic and clinical intent.
• Multimodal embeddings from text and image inputs are fused using a learnable attention mechanism to guide expert routing and enhance representation learning.
• A custom small-scale dataset for healthy food recommendations is constructed using patient descriptions and image inputs to evaluate the model.
• The model is trained end-to-end using cross-entropy and contrastive loss to optimize for precision, recall, NDCG, and MAP@5.
• Image inputs are processed via a vision encoder, but their contribution is found to be limited due to reclassification instability and low-quality inputs.

Experimental results

Research questions

• RQ1 How does combining Mixture-of-Experts (MOE) with large language models (LLMs) improve recommendation performance in healthcare settings?
• RQ2 To what extent do image inputs enhance personalized recommendations, especially in cold start scenarios?
• RQ3 How do image quality and reclassification errors affect the reliability and performance of multimodal recommendation systems?
• RQ4 Can a hybrid MOE-LLM architecture outperform standalone MOE or LLM-based recommendation models in key metrics like precision and NDCG?
• RQ5 What are the practical limitations of integrating visual data into healthcare recommendation systems?

Key findings

• The hybrid MOE-LLM model significantly outperforms both standalone MOE and LLM baselines on all evaluated metrics, including Precision, Recall, NDCG, and MAP@5.
• Image data contributed relatively little to performance improvement, particularly in cold start scenarios where patient descriptions were the primary input.
• Image reclassification issues—especially with low-quality or appearance-changed images—adversely affected recommendation accuracy and stability.
• The model demonstrated robust personalization capabilities, with LLMs effectively interpreting nuanced patient descriptions to generate relevant recommendations.
• Despite the limited gains from images, the multimodal fusion mechanism improved overall representation learning and expert routing efficiency.
• The architecture is scalable and effective for real-world healthcare applications, particularly in domains like personalized nutrition.