[Paper Review] Robust Semantic Communications with Masked VQ-VAE Enabled Codebook

Qiyu Hu, Guangyi Zhang+4 more|arXiv (Cornell University)|Jun 8, 2022

Wireless Signal Modulation Classification4 citations

TL;DR

This paper proposes a robust end-to-end semantic communication framework to combat semantic noise—misinterpretations arising during encoding, transmission, or decoding—by integrating adversarial training with weight perturbation, masked VQ-VAE with noise-aware masking, and a feature importance module (FIM) that suppresses task-irrelevant, noise-sensitive features. The method achieves significant robustness gains with reduced transmission overhead across multiple downstream tasks, including image retrieval and reconstruction, especially under low SNR and semantic noise conditions.

ABSTRACT

Although semantic communications have exhibited satisfactory performance for a large number of tasks, the impact of semantic noise and the robustness of the systems have not been well investigated. Semantic noise refers to the misleading between the intended semantic symbols and received ones, thus cause the failure of tasks. In this paper, we first propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise. In particular, we analyze sample-dependent and sample-independent semantic noise. To combat the semantic noise, the adversarial training with weight perturbation is developed to incorporate the samples with semantic noise in the training dataset. Then, we propose to mask a portion of the input, where the semantic noise appears frequently, and design the masked vector quantized-variational autoencoder (VQ-VAE) with the noise-related masking strategy. We use a discrete codebook shared by the transmitter and the receiver for encoded feature representation. To further improve the system robustness, we develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features. Thus, the transmitter simply needs to transmit the indices of these important task-related features in the codebook. Simulation results show that the proposed method can be applied in many downstream tasks and significantly improve the robustness against semantic noise with remarkable reduction on the transmission overhead.

Motivation & Objective

• To address the lack of systematic study on semantic noise in deep learning-based semantic communication systems.
• To model and classify semantic noise into sample-dependent and sample-independent types, reflecting noise arising from encoding, channel, or decoding stages.
• To develop a robust semantic communication framework that maintains high performance under semantic noise, especially in low-SNR or adversarial environments.
• To reduce transmission overhead by transmitting only indices of task-relevant features via a discrete codebook.
• To enhance system robustness through a feature importance module (FIM) that suppresses noise-related and task-irrelevant features.

Proposed method

• Adversarial training with weight perturbation is used to incorporate samples with semantic noise into the training dataset, improving generalization.
• A masked vector quantized variational autoencoder (VQ-VAE) is designed with a noise-related masking strategy that selectively masks input regions where semantic noise commonly occurs.
• A shared discrete codebook is used at both transmitter and receiver to represent encoded features, enabling efficient index-based transmission.
• The feature importance module (FIM) dynamically suppresses activation of features that are noise-sensitive or irrelevant to the task, enhancing robustness.
• A semantic similarity regularization term is introduced in the loss function to promote orthogonality among codebook vectors, improving representational capacity.
• The system is trained end-to-end to optimize for downstream tasks such as image retrieval and reconstruction, with performance evaluated under varying semantic noise levels and SNR.

Experimental results

Research questions

• RQ1 How does semantic noise—defined as misalignment between intended and received semantic meaning—affect the performance of end-to-end semantic communication systems?
• RQ2 What are the distinguishing characteristics of sample-dependent versus sample-independent semantic noise in semantic communication?
• RQ3 Can adversarial training with weight perturbation effectively improve model robustness against semantic noise?
• RQ4 To what extent does noise-aware masking in VQ-VAE enhance system resilience to semantic noise?
• RQ5 How effective is the feature importance module (FIM) in suppressing noise-related and task-irrelevant features to improve robustness and reduce transmission overhead?

Key findings

• The proposed method significantly improves robustness against both white-box and black-box semantic noise, outperforming Transformer-based and ResNet-based models in adversarial settings.
• The feature importance module (FIM) effectively increases activation of task-relevant features (e.g., features 75–100) while suppressing noise-related features (e.g., features 125–200), reducing redundant activations.
• The semantic similarity regularization term in the loss function enhances codebook quality by promoting near-orthogonality among basis vectors, improving representational efficiency.
• With a patch size of 8, the system achieves superior recall@1 performance in image retrieval, especially in low SNR conditions, outperforming conventional JPEG+LDPC even under high semantic noise (ε=0.012).
• Image reconstruction quality is significantly better with patch size 8 compared to 16, maintaining high fidelity under semantic noise, demonstrating the effectiveness of low-compression, high-robustness design.
• The system reduces transmission overhead by transmitting only codebook indices of important features, enabling efficient communication without sacrificing task accuracy.