Atmospheric Retrievals in a Modern Python Framework

<p>Modern Machine Learning (ML) techniques applied in atmospherical modeling rely heavily on two<br>aspects: good quality and good coverage observations. Among others, Satellite<br>Radiometer (SR) measuremens (Radiances or Brightness Temperatures) offer an excellent trade off<br>between such aspects; moreover SR observations have been providing quite stable Fundamental Cli-<br>mate Data Records (FCDR) for years and are expected to continue to do so in the following decades.<br>This work presents a framework for SR retrievals that uses modern ML standard packages from<br>the Scipy and Pangeo ecosystems; moreover, our retrieval scheme leverage the powerful<br>capabilites provided by NWPSAF’s RTTOV and its Python wrapper.<br>In terms of retrievals we stand on the shoulders of Bayesian Estimation by using Optimal Estima-<br>tion (OE), popularized by Rodgers for 1D atmospherical retrievals; we use pyOpEst<br>which is an open source package developed by Maahn. PyOptimalEstimation is structured<br>following an Object Oriented design, which makes it portable and highly maintainable.</p><p>The contribution presented here ranges from the scientific software design aspects, algorithmic<br>choices, open source contributions, processing speed and scalability; furthermore, simple but effi-<br>cient techniques such as cross-validation were used to evaluate different metrics; for initial test-<br>ing we have used NWPSAF’s model data and observation error covariances from SR literature.</p><p>The open source and community development philosophy are two pillars of this work. Open source<br>allows a transparent, concurrent and continuous development while community development brings<br>together domain experts, software developers and scientists in general; these two ideas allow us to<br>both profit from already developed and well supported tools (e.g. Scipy and Pangeo) and contribute<br>for others whose applications might benefit. This methodology has been successfully used all over the<br>Data Science and ML universe and we believe that the Earth Observation (EO) community would highly benefit in terms of streamlining development and benchmarking of new solutions. Practical examples of success can be found in the Pytroll community.</p><p>Our work in progress is directly linked to present and near future requirements by Earth Observa-<br>tion, in particular the incoming SR streams of data (for operational purposes) is increasing fast<br>and by orders of magnitude. Missions like the EUMETSAT Polar System-Second Generation (EPS-<br>SG, 2023) or the Copernicus Microwave Imager Radiometer (CIMR, 2026) will require scalability<br>and flexibility from the tools to digest such flows of data. We will discuss and show how operational<br>tools can take advantage of the enormous community based developments and standards and become<br>game changers for EO.</p>