[Paper Review] Wireless Semantic Communications for Video Conferencing

Peiwen Jiang, Chao-Kai Wen+2 more|arXiv (Cornell University)|Apr 16, 2022

IoT and Edge/Fog Computing4 citations

TL;DR

This paper proposes a semantic video conferencing system (SVC) that transmits facial keypoints instead of full video frames, reducing bandwidth use while preserving high resolution. By integrating an incremental redundancy HARQ framework with a semantic error detector that leverages video fluency and identity consistency, the system achieves robust performance under varying channel conditions, with CSI feedback further enhancing bit allocation and efficiency.

ABSTRACT

Video conferencing has become a popular mode of meeting even if it consumes considerable communication resources. Conventional video compression causes resolution reduction under limited bandwidth. Semantic video conferencing maintains high resolution by transmitting some keypoints to represent motions because the background is almost static, and the speakers do not change often. However, the study on the impact of the transmission errors on keypoints is limited. In this paper, we initially establish a basal semantic video conferencing (SVC) network, which dramatically reduces transmission resources while only losing detailed expressions. The transmission errors in SVC only lead to a changed expression, whereas those in the conventional methods destroy pixels directly. However, the conventional error detector, such as the cyclic redundancy check, cannot reflect the degree of expression changes. To overcome this issue, we develop an incremental redundancy hybrid automatic repeat-request (IR-HARQ) framework for the varying channels (SVC-HARQ) incorporating a novel semantic error detector. The SVC-HARQ has flexibility in bit consumption and achieves good performance. In addition, SVC-CSI is designed for channel state information (CSI) feedback to allocate the keypoint transmission and enhance the performance dramatically. Simulation shows that the proposed wireless semantic communication system can significantly improve the transmission efficiency.This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Motivation & Objective

• Address the high bandwidth consumption of conventional video conferencing, especially on mobile networks.
• Overcome the limitation of existing semantic video systems that neglect physical layer error impacts on keypoint transmission.
• Design a robust transmission framework that preserves visual quality despite channel-induced keypoint errors.
• Introduce a semantic-aware error detection mechanism that reflects perceptual impact rather than bit-level errors.
• Enhance system adaptability and efficiency through CSI feedback and incremental redundancy HARQ

Proposed method

• Establish a three-level semantic video conferencing (SVC) framework that transmits only facial keypoints to represent motion, enabling high-resolution output with minimal data.
• Develop an incremental redundancy hybrid automatic repeat request (IR-HARQ) framework (SVC-HARQ) with ACK feedback to retransmit incremental redundancy when errors are detected.
• Design a novel semantic error detector that evaluates frame fluency and identity consistency to determine if retransmission is needed, replacing traditional CRC-based detection.
• Introduce SVC-CSI, a CSI feedback mechanism that allocates more bits to subchannels with higher SNR, optimizing bit consumption and performance.
• Train the system end-to-end using deep learning to jointly optimize keypoint extraction, modulation, and error resilience under varying channel conditions.
• Use a combination of identity classification and temporal fluency metrics to detect semantic degradation caused by keypoint errors

Experimental results

Research questions

• RQ1 How does semantic error propagation in keypoint-based video conferencing differ from conventional pixel-level error propagation?
• RQ2 Can a semantic-aware error detector based on video fluency and identity consistency outperform traditional bit-error detection in HARQ systems?
• RQ3 To what extent does CSI feedback improve bit allocation and system performance in semantic video conferencing under fading channels?
• RQ4 How does the proposed SVC-HARQ framework maintain performance under mismatched channel conditions compared to conventional HARQ?
• RQ5 What is the trade-off between performance, bit consumption, and robustness when combining CSI feedback with HARQ in semantic video transmission?

Key findings

• The proposed SVC system achieves a significant compression ratio by transmitting only facial keypoints, preserving high-resolution output while losing only fine facial expressions.
• Transmission errors in SVC primarily alter keypoint locations, causing unnatural facial expressions, but these are often perceptually acceptable due to semantic robustness.
• The semantic error detector effectively identifies frames with reduced fluency and identity inconsistency, enabling targeted retransmission with improved perceptual quality.
• SVC-HARQ achieves better performance than conventional HARQ under varying bit error rates (BER), with Ploss values of 0.186 at BER=0 and 0.207 at BER=0.10, demonstrating robustness.
• SVC-CSI with SNR-based bit allocation improves performance over non-CSI schemes, especially in low-SNR environments, with transmit power dynamically adjusted across subchannels.
• SVC-CSI-HARQ maintains or improves performance under mismatched channels, outperforming standard SVC-HARQ when BER is between 0.02 and 0.10, and showing superior robustness in dynamic environments.