Federated AI-Cubes: Towards Democratizing Big Earth Datacube Analytics

Datacubes are an acknowledged cornerstone for analysis-ready Big Earth Data as they allow more intuitive, powerful services than zillions of "scenes". By abstracting from technical pains they offer two main advantages: for users, it gets more convenient; servers can dynamically optimize, orchestrate, and distribute processing. We propose a combination of datacube service enhancements which we consider critical for making data exploitation more open to non-experts and more powerful, summarized as "Federated AI-Cubes": Location-transparent federation allows users and tools to perceive all datacube assets as a single dataspace, making distributed data fusion a commodity. Instrumental for this is automatic data homogenization performed at import and at query time, based on the open Coverage standards.High-level datacube query languages, such as SQL/MDA and ISO/OGC WCPS, simplify analysis and open up data exploitation to non-programmers. Server-side optimization can automatically generate the individually best distributed workflow for every incoming query. At the same time, queries document workflows without low-level technical garbage, making them reproducible. The seamless integration of AI into datacube analytics plus AI-assisted query writing open up new opportunities for zero-coding exploitation. By not hardwiring a particular model a platform for easy-to-use model sharing emerges. Model Fencing, a new research direction, aims at enabling the server to estimate accuracy of ML model inference embedded in datacube queries. Standards-based interoperability allows users to remain in the comfort zone of their well-known clients, from map browsing over QGIS and ArcGIS up to openEO, R, and python frontends.Cloud/edge integration opens up opportunities for seamless federation of data centers with moving data sources, such as satellites, including flexible onboard processing. In summary, these capabilities together have potential for empowering non-experts and making experts more productive, ultimately democratizing Big Earth Data exploitation and widening Open Science. In our talk, we discuss these techniques based on their implementation in the rasdaman Array DBMS, the pioneer datacube engine, which is operational on multi-Petabyte global assets contributed by research centers in Europe, USA, and Asia. We present challenges and results, supported by live demos many of which are public. Additionally, being editor of the OGC and ISO coverage standards suite, we provide an update on recent progress and future developments. This research is being co-funded by the European Commission through EFRE projects FAIRgeo and SkyFed.