Research on compression technology of meteorological big data based on satellite communication

Abstract Due to communication satellites are limited by word count and frequency, they cannot meet the needs of ocean-going ships for big data of numerical weather prediction. Therefore, a big data compression algorithm combining lossy compression and lossless compression is proposed in this paper. Lossy compression is based on the premise of preserving the core meteorological characteristics of different scale weather systems, and the dimensionality reduction plane octree coding is designed to eliminate redundancy. Lossless compression technology based on triple matrix pointer symmetric cryptographic table is designed on the premise of not changing the data volume. The calculation results of ECMWF numerical weather prediction data from July to December 2021 show that the average lossless compression rate is 0.21%, the average lossless compression rate is 5.28%, the average total compression rate is 0.112‰, and the single data in the compressed region (0°N-45°N, 90°E-135°E) is less than 1.7Kb. Using Tiantong satellite as the communication medium, further tests conducted on Yuezhanyuke No.2 research vessel during July 19–25, 2021 show that it takes 5.2 seconds and 166 seconds on average to complete the ocean-going data transmission of regional and global single compressed data sets, and 2.4 hours to complete the ocean-going data transmission of global compressed data sets for 10 days. In addition, after calculation, the above data can be converted into 1, 32 and 1664 pieces of Beidou-3 short message data respectively. It can be seen that the algorithm has the characteristics of small compressed data storage space and fast transmission speed.