[Paper Review] High-Cardinality Hybrid Shaping for 4D Modulation Formats in Optical Communications Optimized via End-to-End Learning

Vinícius Oliari, Boris Karanov+7 more|arXiv (Cornell University)|Dec 20, 2021

Optical Network Technologies11 citations

TL;DR

This paper proposes a 4D 10 bits/symbol modulation format optimized via end-to-end deep learning using a split-step Fourier method (SSFM) fiber channel model. The autoencoder-based approach jointly optimizes probabilistic and geometric shaping, achieving a 13.6% reach increase over PM-32QAM at 400 Gbps with 20% FEC overhead, demonstrating superior nonlinear tolerance and spectral efficiency without energy constraints.

ABSTRACT

In this paper we carry out a joint optimization of probabilistic (PS) and geometric shaping (GS) for four-dimensional (4D) modulation formats in long-haul coherent wavelength division multiplexed (WDM) optical fiber communications using an auto-encoder framework. We propose a 4D 10 bits/symbol constellation which we obtained via end-to-end deep learning over the split-step Fourier model of the fiber channel. The constellation achieved 13.6% reach increase at a data rate of approximately 400 Gbits/second in comparison to the ubiquitously employed polarization multiplexed 32-QAM format at a forward error correction overhead of 20%.

Motivation & Objective

• Address the challenge of joint optimization of probabilistic shaping (PS) and geometric shaping (GS) in 4D optical modulation formats for long-haul coherent WDM systems.
• Overcome limitations of conventional PS and GS methods that rely on simplified channel models or constant-energy constellations.
• Develop a high-capacity 4D modulation format with 10 bits/symbol for improved spectral efficiency and nonlinear tolerance in nonlinear fiber channels.
• Validate the performance of the learned constellation using generalized mutual information (GMI) and real-world fiber propagation models.
• Demonstrate that end-to-end deep learning over the SSFM channel model enables superior constellation design compared to standard QAM formats.

Proposed method

• Employ an autoencoder framework with separate neural networks for probabilistic shaping (PS) and geometric shaping (GS), trained end-to-end over a nonlinear fiber channel.
• Use the split-step Fourier method (SSFM) as the accurate physical layer model for dual-polarization fiber links, simulating nonlinear impairments including Kerr nonlinearity and dispersion.
• Optimize the 4D constellation and symbol probabilities jointly using backpropagation through the SSFM, enabling direct learning of nonlinear channel effects.
• Implement pulse shaping with a root-raised cosine filter (0.01 roll-off) and upsample to 16 samples per symbol for waveform generation.
• Use generalized mutual information (GMI) as the performance metric to evaluate spectral efficiency and reliability across multiple spans.
• Train the autoencoder to minimize bit error rate by learning optimal constellation points and symbol probabilities that maximize mutual information under nonlinear impairments.

Experimental results

Research questions

• RQ1 Can end-to-end deep learning over an accurate SSFM-based fiber channel model jointly optimize geometric and probabilistic shaping for 4D modulation formats?
• RQ2 Does a 4D 10 bits/symbol constellation learned via autoencoder outperform standard PM-32QAM and PM-64QAM in terms of transmission reach and spectral efficiency?
• RQ3 What is the impact of removing energy constraints on the learned constellation’s performance in nonlinear fiber links?
• RQ4 How does the learned constellation compare to PM-PS64QAM in terms of reach and FEC overhead at high data rates?
• RQ5 Can the optimized constellation achieve better nonlinear tolerance than conventional QAM formats without requiring complex binary labeling schemes?

Key findings

• The proposed 4D 10 bits/symbol constellation achieved a 13.6% reach increase over PM-32QAM at 400 Gbps with 20% FEC overhead, demonstrating superior nonlinear tolerance.
• At 4000 km (50 spans), the optimized constellation achieved an average GMI of approximately 8 bits/symbol, yielding a spectral efficiency of 7.76 bits/s/Hz.
• The constellation outperformed PM-32QAM by 0.3 bits/symbol at optimal launch power and also showed better performance in the linear regime.
• The symbol probabilities converged to a uniform distribution, indicating that geometric shaping was the primary driver of performance gains, not probabilistic shaping.
• The constellation exhibited wide energy variation across symbols, contradicting the common assumption of constant-energy constellations in prior work.
• At 6000 km, the optimized constellation required only 33% FEC overhead to achieve a net rate comparable to PM-64QAM (44% overhead), while gaining 520 km (8.7%) reach over PM-64QAM and 340 km (5.5%) over PM-PS64QAM.