[Paper Review] Nonlinear Trellis Description for Convolutionally Encoded Transmission Over ISI-channels with Applications for CPM

Fabian Schuh, Johannes B. Huber|arXiv (Cornell University)|May 31, 2012

Advanced Wireless Communication Techniques12 references3 citations

TL;DR

This paper proposes a matched decoding scheme using a nonlinear trellis encoder to jointly model convolutional encoding and intersymbol interference (ISI) channels, significantly reducing the number of states in the trellis for coded continuous phase modulation (CPM) without performance loss. By merging the encoder and ISI channel into a single non-linear binary trellis, the method enables efficient maximum-likelihood sequence estimation with reduced complexity, especially under non-coherent reception with differential detection.

ABSTRACT

In this paper we propose a matched decoding scheme for convolutionally encoded transmission over intersymbol interference (ISI) channels and devise a nonlinear trellis description. As an application we show that for coded continuous phase modulation (CPM) using a non-coherent receiver the number of states of the super trellis can be significantly reduced by means of a matched non-linear trellis encoder.

Motivation & Objective

• To reduce the state complexity of super trellis decoding in convolutionally encoded transmission over ISI channels.
• To enable efficient matched decoding for non-coherent CPM by integrating the convolutional encoder and ISI channel into a single nonlinear trellis.
• To demonstrate that state reduction is possible without performance degradation, especially in non-coherent CPM systems with differential detection.
• To compare the proposed method with conventional super-trellis decoding (STD), decision-feedback sequence estimation (DFSE), and BCJR algorithms in terms of complexity and bit error rate (BER).
• To evaluate the impact of noise whitening filters and ISI length on receiver complexity and performance.

Proposed method

• The paper models the ISI channel and convolutional encoder as a single nonlinear binary trellis encoder by combining the binary outputs of the encoder with the M-ary ISI response.
• It uses a modified trellis structure where the M-ary channel impulse response is decomposed into log₂(M) parallel binary convolutions, enabling a unified non-linear trellis description.
• The approach employs a matched decoding (MD) strategy that jointly decodes the convolutional code and the ISI channel using the Viterbi algorithm on the reduced-state trellis.
• A recursive systematic soft-output Viterbi algorithm (RSSE) is applied to further reduce complexity by partitioning the state space based on decision-feedback-like structure.
• The method incorporates noise whitening filters to mitigate spectral shaping effects in non-coherent CPM, while maintaining compatibility with differential detection and matched filtering.
• Monte Carlo simulations are used to compare BER performance across different receiver configurations, including STD, DFSE, BCJR, and the proposed MD/RSSE.

Experimental results

Research questions

• RQ1 Can the super trellis complexity for convolutionally encoded transmission over ISI channels be reduced without performance loss?
• RQ2 How does the proposed nonlinear trellis encoder compare to conventional super-trellis decoding in terms of BER and state count?
• RQ3 To what extent can receiver complexity be reduced using matched decoding with recursive systematic soft-output Viterbi estimation (RSSE)?
• RQ4 How does the length of the ISI channel and the presence of noise whitening filters affect the performance-complexity trade-off?
• RQ5 Can the proposed method be effectively applied to non-coherent CPM systems with differential detection?

Key findings

• The proposed matched decoding (MD) achieves the same bit error rate (BER) performance as conventional super-trellis decoding (STD) but with significantly fewer states—e.g., 8 states instead of 1024 in one simulation.
• For non-coherent CPM with M=4 and 3-RC pulse shaping, MD with 8-state trellis outperforms STD with 1024 states and matches the performance of DFSE with 64 states.
• When using RSSE with DFSE-like partitioning, the receiver complexity can be reduced to as low as 4 states while achieving BER = 10⁻³, outperforming systems with more encoder states.
• The performance gain from increasing ISI length (e.g., L=10) is greater than that from increasing encoder state count, indicating that longer ISI channels are more beneficial than higher-rate encoders.
• The method reduces complexity by up to 99% compared to STD, with MD achieving BER = 10⁻³ at 14 dB Eb/N0 using only 8 states, while STD requires 1024 states.
• The approach is incompatible with bit interleaving, as it relies on direct concatenation of encoder and channel states, limiting its use in interleaved coded modulation schemes.