[Paper Review] Large AI Model Empowered Multimodal Semantic Communications

Feibo Jiang, Li Dong+5 more|arXiv (Cornell University)|Sep 3, 2023

Multimodal Machine Learning Applications5 citations

TL;DR

This paper proposes a Large AI Model-powered Multimodal Semantic Communication (LAM-MSC) framework that unifies text, audio, image, and video transmission using a single semantic model. By leveraging a Multimodal Language Model (MLM) for semantic alignment, a personalized Large Language Model-based Knowledge Base (LKB) to resolve semantic ambiguity, and Conditional GAN-based Channel Estimation (CGE) for fading mitigation, the framework achieves high semantic fidelity with 98.7% transmission accuracy at 25 dB SNR and reduces data size by over 99% compared to conventional methods.

ABSTRACT

Multimodal signals, including text, audio, image, and video, can be integrated into Semantic Communication (SC) systems to provide an immersive experience with low latency and high quality at the semantic level. However, the multimodal SC has several challenges, including data heterogeneity, semantic ambiguity, and signal distortion during transmission. Recent advancements in large AI models, particularly in the Multimodal Language Model (MLM) and Large Language Model (LLM), offer potential solutions for addressing these issues. To this end, we propose a Large AI Model-based Multimodal SC (LAM-MSC) framework, where we first present the MLM-based Multimodal Alignment (MMA) that utilizes the MLM to enable the transformation between multimodal and unimodal data while preserving semantic consistency. Then, a personalized LLM-based Knowledge Base (LKB) is proposed, which allows users to perform personalized semantic extraction or recovery through the LLM. This effectively addresses the semantic ambiguity. Finally, we apply the Conditional Generative adversarial network-based channel Estimation (CGE) for estimating the wireless channel state information. This approach effectively mitigates the impact of fading channels in SC. Finally, we conduct simulations that demonstrate the superior performance of the LAM-MSC framework.

Motivation & Objective

• To address the challenges of data heterogeneity, semantic ambiguity, and signal fading in multimodal semantic communication (SC) systems.
• To unify multimodal data (text, audio, image, video) into a single semantic communication model, eliminating the need for multiple unimodal SC systems.
• To enhance semantic fidelity by resolving ambiguity through personalized knowledge bases derived from large language models (LLMs).
• To improve robustness over fading wireless channels using conditional generative adversarial networks for accurate channel state information (CSI) estimation.
• To demonstrate the superiority of the proposed LAM-MSC framework through end-to-end simulations across diverse multimodal datasets.

Proposed method

• Proposes a Multimodal Alignment (MMA) mechanism using a Multimodal Language Model (MLM) to map multimodal inputs into a unified semantic space while preserving semantic consistency.
• Introduces a personalized Large Language Model-based Knowledge Base (LKB) that enables user-specific semantic extraction and recovery, reducing ambiguity from diverse user backgrounds.
• Employs Conditional Generative Adversarial Networks (CGAN) for Channel Estimation (CGE) to estimate CSI accurately in fading channels, improving transmission reliability.
• Uses a crossed-training strategy that alternately freezes and fine-tunes the semantic and channel models to ensure joint optimization and convergence.
• Applies BERT and cosine similarity as a semantic evaluation metric to quantify the similarity between original and recovered semantic representations.
• Sets a cosine similarity threshold of 0.6 to define semantic correctness, with transmission accuracy calculated as the ratio of correctly transmitted samples.

Experimental results

Research questions

• RQ1 How can multimodal data (text, audio, image, video) be efficiently unified under a single semantic communication model to reduce system overhead?
• RQ2 To what extent can a Multimodal Language Model (MLM) preserve semantic consistency during transformation between multimodal and unimodal data?
• RQ3 How effective is a personalized Large Language Model-based Knowledge Base (LKB) in resolving semantic ambiguity caused by differing user knowledge backgrounds?
• RQ4 Can Conditional GAN-based Channel Estimation (CGE) effectively mitigate the impact of fading channels in semantic communication systems?
• RQ5 What is the achievable transmission accuracy and data reduction gain of the proposed LAM-MSC framework under realistic wireless conditions?

Key findings

• The LAM-MSC framework achieves a transmission accuracy of 98.7% at 25 dB SNR, significantly outperforming lower SNR conditions.
• Audio data achieves the highest accuracy due to lower intrinsic complexity, while video data has the lowest accuracy due to higher data complexity.
• Transmission accuracy decreases as the cosine similarity threshold increases, with a threshold of 0.6 used to define semantic correctness.
• The framework reduces the total data size from 3,114,800 bits (video) to just 32,768 bits by transmitting only semantic representations, achieving over 99% reduction.
• The crossed-training strategy successfully converges both the semantic and channel models, ensuring joint optimization and improved system performance.
• The use of BERT and cosine similarity enables reliable, quantitative evaluation of semantic fidelity between original and recovered data.