Low-Complexity Detection of M-ary PSK Faster-than-Nyquist Signaling

Faster-than-Nyquist (FTN) signaling is a promising non-orthogonal physical layer transmission technique to improve the spectral efficiency of future communication systems but at the expense of intersymbol-interference (ISI). In this paper, we investigate the detection problem of FTN signaling and formulate the sequence estimation problem of any M-ary phase shift keying (PSK) FTN signaling as an optimization problem that turns out to be non-convex and nondeterministic polynomial time (NP)-hard to solve. We propose a novel algorithm, based on concepts from semidefinite relaxation (SDR) and Gaussian randomization, to detect any M-ary PSK FTN signaling in polynomial time complexity regardless of the constellation size M or the ISI length. Simulation results show that the proposed algorithm strikes a balance between the achieved performance and the computational complexity. Additionally, results show the merits of the proposed algorithm in improving the spectral efficiency when compared to Nyquist signaling.


page 1

page 2

page 3

page 4


Low-Complexity Detection for Faster-than-Nyquist Signaling based on Probabilistic Data Association

In this paper, we investigate the sequence estimation problem of faster-...

A Novel Low Complexity Faster-than-Nyquist (FTN) Signaling Detector for Ultra High-Order QAM

Faster-than-Nyquist (FTN) signaling is a promising non-orthogonal pulse ...

Ranking Bracelets in Polynomial Time

The main result of the paper is the first polynomial-time algorithm for ...

Enabling Quality-Driven Scalable Video Transmission over Multi-User NOMA System

Recently, non-orthogonal multiple access (NOMA) has been proposed to ach...

Fresh, Fair and Energy-Efficient Content Provision in a Private and Cache-Enabled UAV Network

In this paper, we investigate a private and cache-enabled unmanned aeria...

Deep Learning-based List Sphere Decoding for Faster-than-Nyquist (FTN) Signaling Detection

Faster-than-Nyquist (FTN) signaling is a candidate non-orthonormal trans...

Improved Linear Embeddings via Lagrange Duality

Near isometric orthogonal embeddings to lower dimensions are a fundament...

I Introduction

In the last decades, we witnessed an exponential growth of wireless traffic that is expected to continue increasing in future communication systems. That said, improving the spectral efficiency (SE) becomes crucial given the scarcity of the available spectrum. Faster-than-Nyquist (FTN) signaling [1] is one of the candidate solutions to increase the SE of future communication networks. In FTN signaling, the pulses’ transmission rate exceeds the Nyquist limit, and this results in intersymbol-interference (ISI) between the received pulses. James Mazo [2] proved that the minimum distance between the received pulses will not be reduced if the signaling rate, that exceeds the Nyquist limit, is within what is known as Mazo limit (sinc pulses can be packed up to 0.802 of its symbol duration). There were doubts raised by Foschini about the benefits of FTN signal in [3]; however, it potentials were revealed in [4, 5] when Rusek and Anderson showed that lower bounds on the information rate of FTN signalling are in most cases higher than the information rates of conventional Nyquist signalling. Thenceforth, the concept of FTN signalling has been extended in many directions: different pulse shapes [6], frequency-domain [7, 8], non-binary signalling [9, 10], non-orthogonal pulses [11], and multiple-input-multiple-output channels [12].

Low-complexity detection of FTN signaling attracted the attention of research community in recent years. For instance, the authors in [13] proposed a technique to reduce the search space of the maximum likelihood sequence estimation (MLSE) algorithm, and hence, attain the optimal performance at reduced complexity. Suboptimal solutions of the MLSE detection problem were reported in [14, 15], where reduced state or reduced search trellis coding-based Bahl-Cocke-Jelinek-Raviv (BCJR) algorithms were proposed, respectively. The computational complexity of the works in [13, 16, 14, 15] are still exponential with the ISI length, and extending them to high-order modulation will be problematic. In [17], the authors proposed a polynomial-time complexity sub-optimal detection algorithm of any high-order quadrature amplitude modulation (QAM) FTN signaling. A novel FTN detector for binary phase shift keying (BPSK) and quadrature PSK (QPSK) on a symbol-by-symbol basis is proposed in [18], where the computational complexity is independent of the ISI length.

-ary phase shift keying (PSK) modulates only the phase of symbols, and hence, the transmitter can always operate at peak power (unlike -ary QAM). This is why -ary PSK is preferred in constant amplitude transmission for satellite communications [19]. The benefits of employing FTN for satellite communications were recently explored in [20]. That said, this paper investigates the detection problem of -ary phase shift keying (PSK) FTN signaling. We formulate the MLSE problem of any -ary PSK FTN signaling as an optimization problem that turns out to be non-convex and nondeterministic polynomial time (NP)-hard to solve. We used concept of semidefinite relaxation to produce a number of relaxed convex optimization problems, where the global optimal solution can be reached in polynomial time complexity regardless of the constellation size or the ISI length. Then, with the help of Gaussian randomization, we obtain a feasible solution to the original MLSE of -ary PSK FTN signaling problem from the the relaxed convex problem solution. We present simulation results to validate our theoretical analysis.

The paper is organized as follows. The -ary PSK FTN signaling system model and the MLSE detection problem are discussed in Section II; while the proposed SDR technique and Gaussian randomization are investigated in Section III. The numerical results are presented in Section IV and finally the paper is concluded in Section V.

Ii System Model and FTN Signaling Detection Problem Formulation

Fig. 1: Block diagram of FTN signaling.

Figure 1 shows the block diagram of a baseband communication system employing FTN signaling. At the transmitter, the data symbols are drawn from a -ary PSK constellation and then shaped by a unit-energy pulse . We transmit a total of symbols, each every , where is the time packing factor and is the symbol duration. The received signal – after being affected by additive white Gaussian noise (AWGN) channel – is passed through a filter matched to the transmit pulse, and is given as


where is the complex -ary PSK data symbol, is the data symbol energy, , with

is the AWGN with zero mean and variance

, and is the signaling rate. The output signal of the matched-filter is sampled every and can be expressed as

We rewrite (II

) in a vector form as


where , , , and are the transmit data symbol vector, ISI vector, colored noise vector, and the convolution operator, respectively. One can see from (II) that the noise samples are correlated, and in this paper, we follow the approach of designing a whitening matched filter at the receiver of the FTN signaling [16]. In particular, we use spectral factorization in order to find the approximate whitening filter coefficients from the vector . This is achieved by factorizing into [21], where

is the z-transform of

. For more details on the design of the whitened matched filter for FTN receivers, we refer the reader to [17, 16]. Accordingly, the received signal in (3), after passing through the approximate whitening filter , is given by


where is the white Gaussian noise and the causal ISI is constructed from as . Equation (4) is equivalently expressed as


where is the ISI matrix constructed from the vector . Finally, the MLSE problem to detect -ary PSK FTN signaling is written as

s. t. (6)

where is the -norm. The constraint in (II) denotes that the decision variables are discrete as they are drawn from an -ary PSK constellation. Exhaustive search can be used to solve the MLSE problem in (II); however, it is too complex to be implemented in practical systems. In the following section, we propose a suboptimal reduced-complexity detection algorithm to detect -ary PSK FTN signaling based on concepts from semidefinite relaxation and Gaussian randomization.

Iii Proposed Algorithm to detect -ary PSK FTN Signaling

We propose our semidefinite relaxation based algorithm for the detection problem of any -ary PSK FTN signaling in this section. As it will be discussed, the proposed scheme provides a sub-optimal solution at polynomial time complexity regardless of the constellation size or the ISI length.

The objective of the problem in (II) can be re-written as


where represents the real part of complex numbers and , where denotes the conjugate transpose. Accordingly, the MLSE problem in (II) to detect -ary PSK FTN signaling can be re-formulated as

s. t. (8)

where denotes the set of all complex Hermitian matrices. Please note that the term in (7) is dropped from the objective function in (III) as it is not a function of the decision variables, and hence, is constant. Please note that we define a new variable , which is equivalent to have the matrix being a rank one () Hermitian positive semidefinite (). The problem in (III) is still NP-hard and the main difficulties lie in the non-convex rank one and the discrete constellation constraints. In the following, we investigate and propose techniques to properly address these non-convex constraints.

The objective function in (III) is linear in the matrix and data symbols vector . However, and as discussed earlier, the reformulation from (II) to (III) allows us to reveal one of the difficulties in solving the MLSE optimization problem to detect the -ary PSK FTN signaling. It is clear from (III) that the difficulty lies in the nonconvex rank one constraint. We relax this problem into a convex one by replacing the nonconvex rank one constraint with a convex positive semidefinite constraint, i.e., [22]. Further, we substitute this positive semidefinite constraint, i.e., , with its schur complement [23]. Hence, the semidefinite relaxation problem of detecting -ary FTN signaling is written as

s. t. (9)

The difficulty now in solving (III) lies in the discrete constellation constraint. To tackle such difficulty, we follow a similar approach to the one presented in [24] and relax the discrete constellation constraint (with discrete decision variables) to continuous ones that lie on a unit circle. Such relaxation is guaranteed to achieve a sub-optimal solution within a certain accuracy of the solution of the original MLSE problem, at reduced computational complexity [24]. That said, the problem in (III) is rewritten as

s. t. (10)

It is shown in [25] that the convex problem in (III) can be solved efficiently using numerical solvers. The challenging task is to convert the global solutions and of the convex SDR problem in (III) into a feasible solution to the original -ary PSK FTN signaling in (III). Among several methods that can achieve this task, we use a Gaussian randomization method inspired by the works in [24, 22] due to its efficiency and low complexity. It is worthy to emphasize that Gaussian randomization, as well as other methods used to obtain feasible solution to the original detection problem that is NP-hard, may not achieve the global optimal solution. This is not surprising as we would have solved an NP-hard problem in a polynomial time complexity. Gaussian randomization solves a stochastic version of (III), and the sub-optimal solution to (III) is selected to be the solution that minimizes the objective in (III) among candidate solutions generated from a multivariate Gaussian random vectors with mean and covariance matrix (). The Gaussian randomization procedure can be explained as follows. We generate where is the total number of randomization iterations, as , then we find as follows [24]:

The optimal value of , i.e., , is selected to minimize the objective function in (III) as


and then, the sub-optimal solution vector is identified as


The proposed algorithm111Please note that the proposed algorithm can be extended to produce soft-outputs, and hence, to be used with channel coding. This can be achieved by exploiting Gaussian randomization to efficiently calculate the log-likelihood ratio (LLR). to detect -ary PSK FTN signaling is formally expressed as follows

  1. Input: The inputs to our SDR-based algorithm are the received samples , the whitened matched filter impulse response, pulse shape (ISI matrix ), and the Gaussian randomization number of iterations .

  2. Solve the relaxed convex optimization problem in in (III) and find the optimal solutions ( and ).

  3. Generate random variable

    as , .

  4. Find as in (12).

  5. Calculate candidate suboptimal solutions to the FTN detection problem in (III) as .

  6. Output: The estimated data symbol vector .

– Complexity Analysis:

Solving the -ary PSK FTN signaling detection problem in (III) requires a complexity of [22]; while the complexity of the Gaussian randomization step is [22]. Accordingly, the the total computational complexity of the proposed algorithm is , which is polynomial in the received block length and is independent of the constellation size and the ISI length.

Iv Simulation Results

The performance of our proposed algorithm to detect 8-ary PSK FTN signaling employing a root-raised cosine (rRC) pulse is investigated in this section. We assume a roll-off factor and , unless otherwise mentioned; and the number of iterations of Gaussian randomization is set to 1000.

Iv-a Performance of Our Proposed SDR-based Algorithm to Detect 8-ary FTN Signaling

Fig. 2: 8-ary PSK FTN signaling performance of the proposed SDR-based algorithm for and different values of .

The BER of 8-ary PSK FTN signaling of the proposed SDR-based scheme is shown in Fig. 2. The performance is evaluated for roll-off factor and for spectral efficiency (SE) values of 3.08, 2.89, and 2.71 bits/s/Hz that correspond to values of 0.75, 0.80, and 0.85, respectively. One can notice that at SE of 2.71 bits/s/Hz (), our proposed SDR-based algorithm can completely remove the ISI and approach the BER of Nyquist transmission. This translates to an improvements of SE by 17.64 for the same roll-off factor value, SNR, and bandwidth. Similarly at BER = , it can be seen that improvements in SE by 33.33 and 25 are achieved for and at a penalty of 0.6 and 1.4 dB in the SNR, respectively.

Fig. 3: Effect of the roll-off factor of rRC pulse on the achieved SE by our proposed SDR-based algorithm at BER = .

Fig. 3 investigates the effect of the roll-off factor of the rRC pulse on the achieved SE of our proposed SDR-based algorithm at BER = . For a fair comparison with ISI-free transmission, we vary the value of (starting with a higher value of that is decremented) to maintain the same SNR achieved by the ISI-free transmission at BER = . First, one notices that the improvements of the SE achieved from our algorithm (when compared to its counterpart of ISI-free transmission) increases with increasing the value of . Additionally, the SE achieved by our proposed scheme for is higher than the maximum SE of ISI-free transmission achieved at .

Iv-B Performance Comparison with the State-of-the-art Detectors

Fig. 4: Comparison of our proposed SDR-based algorithm to detect 8-ary PSK FTN signaling and the M-BCJR algorithm in [26] to detect QPSK FTN signaling for and the same SE.

We compare the performance of our proposed SDR-based algorithm to detect 8-ary FTN signaling to its counterpart of the M-BCJR algorithm in [26] to detect QPSK FTN signaling (i.e., both algorithms achieve the same SE of 3.08 bits/s/Hz for ) in Fig. 4. As can be seen, our proposed SDR-based algorithm (which is of polynomial complexity) is approximately at 1 dB SNR difference from the M-BCJR algorithm in [26] (which is of exponential complexity of the ISI length), at BER = . It is evident from Fig. 4 that our proposed SDR-based detection algorithm achieves a suboptimal solution that strikes a balance between complexity and performance.

V Conclusion

FTN signaling showed merits in improving the spectral efficiency of future communication systems; however, the main challenge is the high complexity of its receivers. In this paper, we have proposed a novel algorithm to estimate the transmit data symbols of any -ary PSK FTN signaling. The proposed algorithm is based on concepts of semidefinite relaxation and Gaussian randomization and was shown to have polynomial complexity regardless of the ISI length or the constellation size . Simulation results identified the favourable operating region of our proposed SDR-based scheme. For instance, improvements of the SE up to 33.33 can be achieved with a small increase of the SNR. Additionally, it has been shown that the improvements of the SE increases with increasing the roll-off factor.


  • [1] J. B. Anderson, F. Rusek, and V. Öwall, “Faster-than-Nyquist signaling,” Proc. IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.
  • [2] J. Mazo, “Faster-than-Nyquist signaling,” Bell Syst. Tech. J., vol. 54, no. 8, pp. 1451–1462, Oct. 1975.
  • [3] G. J. Foschini, “Contrasting performance of faster binary signaling with QAM,” Bell Syst. Tech. J., vol. 63, no. 8, pp. 1419–1445, Oct. 1984.
  • [4] F. Rusek and J. B. Anderson, “On information rates for faster than Nyquist signaling,” in Proc. IEEE Global Communication Conference (GLOBECOM), Dec. 2006, pp. 1–5.
  • [5] ——, “Constrained capacities for faster-than-Nyquist signaling,” IEEE Trans. Inf. Theory, vol. 55, no. 2, pp. 764–775, Feb. 2009.
  • [6] A. D. Liveris and C. N. Georghiades, “Exploiting faster-than-Nyquist signaling,” IEEE Trans. Commun., vol. 51, no. 9, pp. 1502–1511, Sep. 2003.
  • [7] F. Rusek and J. B. Anderson, “The two dimensional Mazo limit,” in Proc. IEEE International Symposium on Information Theory (ISIT), Sep. 2005, pp. 970–974.
  • [8] ——, “Multistream faster than Nyquist signaling,” IEEE Trans. Commun., vol. 57, no. 5, pp. 1329–1340, May 2009.
  • [9] C.-K. Wang and L.-S. Lee, “Practically realizable digital transmission significantly below the Nyquist bandwidth,” IEEE Trans. Commun., vol. 43, no. 2-4, pp. 166–169, Feb., Mar., and Apr. 1995.
  • [10] F. Rusek and J. B. Anderson, “Non binary and precoded faster than Nyquist signaling,” IEEE Trans. Commun., vol. 56, no. 5, pp. 808–817, May 2008.
  • [11] N. Seshadri and J. B. Anderson, “Asymptotic error performance of modulation codes in the presence of severe intersymbol interference,” IEEE Trans. Inf. Theory, vol. 34, no. 5, pp. 1203–1216, Sep. 1988.
  • [12] F. Rusek, “On the existence of the Mazo-limit on MIMO channels,” IEEE Trans. Wireless Commun., vol. 8, no. 3, pp. 1118–1121, Mar. 2009.
  • [13] E. Bedeer, H. Yanikomeroglu, and M. H. Ahmed, “Reduced complexity optimal detection of binary faster-than-Nyquist signaling,” in Proc. IEEE International Conference on Communications (ICC), Paris, France, 2017, May 2017, pp. 1–6.
  • [14] J. B. Anderson, A. Prlja, and F. Rusek, “New reduced state space BCJR algorithms for the ISI channel,” in Proc. IEEE International Symposium on Information Theory (ISIT), Jun. 2009, pp. 889–893.
  • [15] A. Prlja and J. B. Anderson, “Reduced-complexity receivers for strongly narrowband intersymbol interference introduced by faster-than-Nyquist signaling,” IEEE Trans. Commun., vol. 60, no. 9, pp. 2591–2601, Sep. 2012.
  • [16] A. Prlja, J. B. Anderson, and F. Rusek, “Receivers for faster-than-Nyquist signaling with and without turbo equalization,” in Proc. IEEE International Symposium on Information Theory, Jul. 2008, pp. 464–468.
  • [17] E. Bedeer, M. H. Ahmed, and H. Yanikomeroglu, “Low-complexity detection of high-order QAM faster-than-Nyquist signaling,” IEEE Access, vol. 5, pp. 14 579–14 588, Jul. 2017.
  • [18] ——, “A very low complexity successive symbol-by-symbol sequence estimator for faster-than-nyquist signaling,” IEEE Access, vol. 5, pp. 7414–7422, Mar. 2017.
  • [19] G. E. Corazza, Digital Satellite Communications.   Springer, 2007.
  • [20] B. F. Beidas, R. I. Seshadri, M. Eroz, and L.-N. Lee, “Faster-than-Nyquist signaling and optimized signal constellation for high spectral efficiency communications in nonlinear satellite systems,” in Proc. IEEE Military Communications Conference (MILCOM), 2014, pp. 818–823.
  • [21] A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing.   Pearson, 2010.
  • [22] Z.-Q. Luo, W.-K. Ma, A. M.-C. So, Y. Ye, and S. Zhang, “Semidefinite relaxation of quadratic optimization problems,” IEEE Signal Process. Mag., vol. 27, no. 3, p. 20, May 2010.
  • [23] S. Boyd and L. Vandenberghe, Convex Optimization.   Cambridge University Press, 2004.
  • [24] A. M.-C. So, J. Zhang, and Y. Ye, “On approximating complex quadratic optimization problems via semidefinite programming relaxations,” Mathematical Programming, vol. 110, no. 1, pp. 93–110, Jun. 2007.
  • [25] M. Grant and S. Boyd, CVX: Matlab Software for Disciplined Convex Programming, 2010. [Online]. Available: cvxr.com/cvx.
  • [26] J. B. Anderson and A. Prlja, “Turbo equalization and an M-BCJR algorithm for strongly narrowband intersymbol interference,” in Proc. IEEE International Symposium on Information Theory and its Applications (ISIT), Jun. 2010, pp. 261–266.