Joint estimation of parameter and state with hybrid data assimilation and machine learning

For parameter and state estimation problems, when observation is sparse and has large error covariance, the estimation results tend to have bias and lead to inaccurate forecasts further. To reduce the bias, we propose to construct a proposal density function with a smaller covariance for the particle filter by conditionally iterating the ensemble transform Kalman filter. First an ensemble is obtained by the ensemble transform Kalman filter. If the distance between the current model forecast (background) and the ensemble members is larger than a predefined threshold then repeats applying the ensemble transform Kalman filter to generate a new model forecast ensemble. Finally, update the weight of ensemble members with the particle filter. We use deep residual neural networks to learn a surrogate model of the assimilation process and combine it with data assimilation method to obtain better forecasts. Experiments results show that our method can effectively reduce the bias compared to the ensemble transform Kalman filter and weighted ensemble transform Kalman filter, especially in case of sparse observations. The results also show that when the observation frequency is low, using a deep residual neural network as surrogate model to generate data for the assimilation process gives more accurate state forecasts than conventional data assimilation method.