[Paper Review] Multi-hop RIS-Empowered Terahertz Communications: A DRL-based Hybrid Beamforming Design

Chongwen Huang, Zhaohui Yang+6 more|arXiv (Cornell University)|Jan 22, 2021

Advanced Wireless Communication Technologies81 references415 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based hybrid beamforming design for multi-hop reconfigurable intelligent surface (RIS)-assisted terahertz (THz) communications to combat severe path loss. By jointly optimizing digital beamforming at the base station and analog beamforming at multiple RISs via DRL, the scheme achieves 50% higher coverage range than benchmarks, demonstrating state-of-the-art performance for NP-hard beamforming in multi-hop THz networks.

ABSTRACT

Wireless communication in the TeraHertz band (0.1--10 THz) is envisioned as one of the key enabling technologies for the future sixth generation (6G) wireless communication systems scaled up beyond massive multiple input multiple output (Massive-MIMO) technology. However, very high propagation attenuations and molecular absorptions of THz frequencies often limit the signal transmission distance and coverage range. Benefited from the recent breakthrough on the reconfigurable intelligent surfaces (RIS) for realizing smart radio propagation environment, we propose a novel hybrid beamforming scheme for the multi-hop RIS-assisted communication networks to improve the coverage range at THz-band frequencies. Particularly, multiple passive and controllable RISs are deployed to assist the transmissions between the base station (BS) and multiple single-antenna users. We investigate the joint design of digital beamforming matrix at the BS and analog beamforming matrices at the RISs, by leveraging the recent advances in deep reinforcement learning (DRL) to combat the propagation loss. To improve the convergence of the proposed DRL-based algorithm, two algorithms are then designed to initialize the digital beamforming and the analog beamforming matrices utilizing the alternating optimization technique. Simulation results show that our proposed scheme is able to improve 50\% more coverage range of THz communications compared with the benchmarks. Furthermore, it is also shown that our proposed DRL-based method is a state-of-the-art method to solve the NP-hard beamforming problem, especially when the signals at RIS-assisted THz communication networks experience multiple hops.

Motivation & Objective

• Address the challenge of severe path loss and limited coverage in terahertz (THz) communications due to high propagation attenuation and molecular absorption.
• Overcome the limitations of single-hop RIS systems by enabling multi-hop transmission using multiple passive, controllable RISs.
• Develop a joint digital and analog beamforming design for multi-user, multi-hop RIS-THz networks to maximize spectral efficiency and coverage.
• Propose a DRL-based algorithm to solve the NP-hard hybrid beamforming problem without requiring explicit channel state information or complex mathematical models.
• Improve convergence and performance through alternating optimization-based initialization of beamforming matrices.

Proposed method

• Formulate a non-convex joint optimization problem for digital beamforming at the base station and analog beamforming at multiple RISs in a multi-hop THz network.
• Design a DRL-based algorithm using actor-critic networks to learn optimal beamforming matrices without prior knowledge of channel models.
• Introduce two initialization methods using alternating optimization to improve DRL convergence: one for digital beamforming and one for analog beamforming at each RIS.
• Use a reward function based on sum rate to train the DRL agent, with state representation including channel state information and system parameters.
• Implement the DRL framework with experience replay and target networks to stabilize training and improve policy learning.
• Scale the DRL framework to various system configurations (e.g., number of RISs, users, power levels) with low implementation complexity.

Experimental results

Research questions

• RQ1 Can a DRL-based approach effectively solve the NP-hard hybrid beamforming problem in multi-hop RIS-empowered THz networks?
• RQ2 How does the proposed DRL-based beamforming design compare to conventional benchmarks in terms of coverage range and spectral efficiency?
• RQ3 What impact do system parameters such as transmission power, RIS element count, and Rician factor have on DRL convergence and performance?
• RQ4 How does the choice of learning rate affect the convergence speed and final reward (sum rate) of the DRL agent?
• RQ5 To what extent can RIS deployment in multiple hops extend the coverage range of THz communications?

Key findings

• The proposed DRL-based hybrid beamforming scheme improves the coverage range of THz communications by 50% compared to benchmark schemes, significantly extending usable transmission distances.
• The algorithm achieves faster convergence when using lower transmission power (e.g., 5W) due to smaller state space, though higher power (30W) yields higher overall rewards and sum rates.
• The average sum rate converges over time steps, with rewards increasing under higher SNR and larger RIS element counts (e.g., 64 elements vs. 32).
• A learning rate of 0.001 yields the best performance, while both higher (0.01) and lower (0.00001) values result in suboptimal convergence and lower rewards.
• The Rician factor has a strong impact: higher values (e.g., 10) that favor line-of-sight (LoS) components lead to significantly higher total throughput compared to non-LoS scenarios (factor = 1).
• The cumulative distribution function (CDF) of average rewards shows that increasing RIS elements (Ni = 64) and transmission power (Pt = 30W) leads to consistently higher performance across system configurations.