Hybrid Constellation Shaping 64QAM Based on Hexagonal Lattice of Constellation Subset

Increasing demand for higher-speed and large-capacity data communications has driven the development of constellation shaping technology. This paper proposes a hybrid constellation shaping scheme for 64-quadrature amplitude modulation (64QAM) based on hexagonal lattice of a constellation subset. The proposed scheme aims to enhance the nonlinear tolerance of higher-order modulated signals and further improve the constellation shaping gain. The initial quantitative characterization of the constellation is firstly performed based on the hexagonal lattice structure. Then, the objective function of maximizing constellation figure of merits (CFM) is utilized to determine the position distribution of constellation points, resulting in the generation of the geometric shaping-64QAM (GS-64QAM) signal. Finally, according to concentric hexagonal layers, all constellation points are divided into multiple subsets where points within the same subset are assigned the same probability, and the hybrid shaping-64QAM (HS-64QAM) signal is generated. To validate the effectiveness of the proposed scheme, the experimental verification was demonstrated in a 120 Gbit/s multi-span coherent optical communication system. Experimental results indicate that, at the soft-decision forward error correction threshold, HS-64QAM achieves an optical signal-to-noise ratio (OSNR) gain of 1.9 dB and 4.1 dB over uniform GS-64QAM in back-to-back and 375 km transmission scenarios, respectively. Furthermore, HS-64QAM achieves an OSNR gain of 2.7 dB and 7.6 dB over uniform Square-64QAM in back-to-back and 375 km transmission scenarios, respectively.