[Paper Review] Semantic Importance-Aware Communications Using Pre-trained Language Models

Shuaishuai Guo, Yanhu Wang+2 more|arXiv (Cornell University)|Feb 12, 2023

COVID-19 diagnosis using AIMedicine3 citations

TL;DR

This paper proposes a semantic importance-aware communication (SIAC) scheme using pre-trained language models (e.g., ChatGPT, BERT) to quantify frame-level semantic importance and enable importance-aware power allocation. By embedding a cross-layer manager with a pre-trained model, SIAC improves semantic reliability in existing systems with minimal modification, achieving lower semantic loss than equal-priority transmission, especially at low SNR.

ABSTRACT

This letter proposes a semantic importance-aware communication (SIAC) scheme using pre-trained language models (e.g., ChatGPT, BERT, etc.). Specifically, we propose a cross-layer design with a pre-trained language model embedded in/connected by the cross-layer manager. The pre-trained language model is utilized to quantify the semantic importance of data frames. Based on the quantified semantic importance, we investigate semantic importance-aware power allocation. Unlike existing deep joint source-channel coding (Deep-JSCC)-based semantic communication schemes, SIAC can be directly embedded into current communication systems by only introducing a cross-layer manager. Our experimental results show that the proposed SIAC scheme can achieve lower semantic loss than existing equal-priority communications.

Motivation & Objective

• To address the incompatibility of deep joint source-channel coding (Deep-JSCC) schemes with existing communication systems due to retraining and system-wide updates.
• To enable semantic communication in legacy systems by introducing only a cross-layer manager, avoiding full system overhaul.
• To quantify semantic importance of data frames using pre-trained language models (e.g., ChatGPT, BERT) for dynamic power allocation.
• To minimize semantic-level loss by prioritizing transmission of semantically important frames through optimized power allocation.

Proposed method

• Integrates a pre-trained language model (e.g., ChatGPT or BERT) into a cross-layer manager to assess semantic importance of data frames.
• Uses ChatGPT to generate messages with highlighted important words, assigning importance weights based on bolded terms.
• Employs BERT to compute semantic importance via sentence embedding similarity, defining semantic loss using Equation (1).
• Applies manifold optimization via the Manopt toolbox to solve the power allocation problem in Equation (4), minimizing expected important word errors or semantic loss.
• Designs two variants: ChatGPT-SIAC for generative importance labeling and BERT-SIAC for embedding-based importance quantification.
• Employs a frame-based transmission model with 20 frames per batch, each containing 5 words, and uses outage probability to model frame error.

Experimental results

Research questions

• RQ1 Can pre-trained language models be effectively used to quantify semantic importance of data frames in communication systems?
• RQ2 How can semantic importance be leveraged to improve reliability in low-SNR or high-loss environments?
• RQ3 Can a cross-layer manager with a pre-trained model be integrated into existing communication systems with minimal modification?
• RQ4 What is the performance gain of importance-aware power allocation over equal-priority transmission in terms of semantic loss and word error rate?

Key findings

• ChatGPT-SIAC achieves the lowest expected important word errors by design, as its objective function directly minimizes such errors.
• BERT-SIAC outperforms equal-priority transmission in semantic loss reduction, especially at low total power (5–10 W), with performance gain diminishing as power increases.
• The proposed SIAC schemes reduce semantic loss compared to equal-priority transmission, with BERT-SIAC showing optimal performance when semantic similarity is the metric.
• ChatGPT generates words at a slightly reduced speed when highlighting important words, indicating a marginal delay increase of less than 10%.
• Self-hosted BERT model processes frames at ~25 frames per minute, which is acceptable only for delay-tolerant applications.
• The implementation code is publicly released on GitHub for reproducibility and further research.