[논문 리뷰] Reinforcement Learning-powered Semantic Communication via Semantic Similarity

Kun Ping Lu, Rongpeng Li외 3명|arXiv (Cornell University)|2021. 08. 27.

Wireless Signal Modulation Classification참고 문헌 60인용 수 44

한 줄 요약

이 논문은 의미 RL(SemanticRL)을 제안합니다. 이는 RL 기반의 의미 전달 프레임워크로, 비트 수준 정확도 대신 의미 유사도 최대화를 목표로 하며, 자기비평 학습과 비분리형 송수신기(SCSIU)를 사용하여 비미분 가능 목표와 채널을 다룹니다. 텍스트 데이터에서 의미 회복 성능이 향상되었고 RL 기반 이미지 전송 시나리오로 확장됩니다.

ABSTRACT

We introduce a new semantic communication mechanism - SemanticRL, whose key idea is to preserve the semantic information instead of strictly securing the bit-level precision. Unlike previous methods that mainly concentrate on the network or structure design, we revisit the learning process and point out the semantic blindness of commonly used objective functions. To address this semantic gap, we introduce a schematic shift that learns from semantic similarity, instead of relying on conventional paired bit-level supervisions like cross entropy and bit error rate. However, developing such a semantic communication system is indeed a nontrivial task considering the non-differentiability of most semantic metrics as well as the instability from noisy channels. To further resolve these issues, we put forward a self-critic reinforcement learning (RL) solution which allows an efficient and stable learning on any user-defined semantic measurement, and take a step further to simultaneously tackle the non-differentiable semantic channel optimization problem via self-critic stochastic iterative updating (SCSIU) training on the decoupled semantic transceiver. We have firstly tested the proposed method in the challenging European-parliament dataset, which confirms the superiority of our method in revealing the semantic meanings, and better handling the semantic noise. Apart from the experimental results, we further provide an in-depth look at how the semantic model behaves, along with its superb generalization ability in real-life examples. An RL-based image transmission extension is also exemplified, so as to prove the generalization ability and motivate future discussion.

연구 동기 및 목표

• Shift the goal from bit-level accuracy to maximizing semantic similarity in communication systems.
• Provide a framework that optimizes non-differentiable semantic similarity metrics as the training objective.
• Address non-differentiable channel effects with a self-critic reinforcement learning approach.
• Introduce a decoupled semantic transceiver variant (SCSIU) to jointly handle semantic encoding and decoding without extra parameters.
• Demonstrate robustness and generalization on real datasets and extend to RL-based image transmission.

제안 방법

• Define semantic similarity as the objective for transceiver optimization, allowing non-differentiable metrics like BLEU and CIDEr to guide learning.
• Use a reinforcement learning paradigm with a policy gradient that optimizes the semantic similarity score Theta(m, m_hat).
• Introduce a self-critic training scheme to reduce gradient variance and enable stable learning without extra baseline networks.
• Propose SemanticRL-JSCC where encoder and decoder are trained with a self-critic policy gradient, including a multinomial sampling for exploration.
• Extend to SemanticRL-SCSIU where the encoder and decoder are decoupled and trained with self-critic updates on continuous (encoder) and discrete (decoder) policies.
• Provide operational details for handling sparse rewards and episodic sequence generation, with equations for returns and gradient estimations.

실험 결과

연구 질문

• RQ1 Can semantic similarity metrics guide end-to-end training of a communication system without differentiable supervision?
• RQ2 How can self-critic reinforcement learning be used to stabilize training for large-scale semantic transmission tasks?
• RQ3 Does optimizing semantic similarity improve high-order semantic alignment compared to bit-level objectives?
• RQ4 Can the approach be extended to a decoupled, large-scale transceiver that handles non-differentiable channels?
• RQ5 How well do BLEU and CIDEr-based objectives perform in practical, real-world datasets and in RL-based image transmission extensions?

주요 결과

• SemanticRL improves the alignment of transmitted meaning by optimizing semantic similarity rather than bit-level accuracy.
• Self-critic training provides low-variance, stable policy gradients for large semantic spaces without extra baseline networks.
• The framework accommodates non-differentiable semantic metrics such as BLEU and CIDEr as optimization targets.
• A decoupled variant (SemanticRL-SCSIU) enables joint or separate optimization of encoder and decoder under non-differentiable channel conditions.
• Experiments on the European-parliament dataset and RL-based image transmission demonstrate robustness and generalization of the semantic-oriented approach.