Practical notes on lossy compression of scientific data

<p>Lossy compression methods are extremely efficient in terms of space and performance and allow for reduction of network bandwidth and disk space needed to store data arrays without sacrificing the number of stored values.  Lossy compression involves an irreversible transformation of data that reduces the information content of the data.  The transformation introduces a distortion that is normally measured in terms of absolute or relative error. The error is higher for higher compression ratios.  A good choice of lossy compression parameters maximizes the compression ratio while keeping the introduced error within acceptable margins.  Negligence or failure to chose a right compression method or its parameters leads to poor compression ratio, or loss of the data.</p><p>A good strategy for lossy compression would involve sepcification of the acceptible error margin and choice of compression parameters and storage format. We will discuss specific techniques of lossy compression, and illustrate pitfalls in choice of the error margins and tools for lossy/lossless compression. The following specific topics will be covered:</p><p>1. Packing of floating-point data to integers in NetCDF is sub-optimal in most cases,   and for some quantities leads to severe errors.<br>2. Keeping relative vs absolute precision: false alternative.<br>3. Acceptible error margin depends on both the origin and the intended application of data.<br>4. Smart algorithms to decide on compression parameters have limited area of applicability,   which has to be considered in each individual case.<br>5. Choice of a format for compressed data (NetCDF, GRIB2, Zarr): tradeoff between size, speed and precision.<br>6. What "number_of_significant_digits" and "least_significant_digit" mean in terms of relative/absolute error.<br>7. Bit-Shuffle is not always beneficial.</p>