An Pangeo/ODC-based backend that runs "out of the box"

[m-mohr]

(This is a discussion, not a vote)

We had quite some interest from the regional ODC initiatives (Swedish, Brazil, Swiss) but we couldn't really offer an easy solution. We then started documenting the steps at https://openeo.org/documentation/1.0/developers/backends/opendatacube.html but it's still combining a lot of different things and it seems not so easy overall. Therefore, the idea came up again today to have someone work on an ODC or Pangeo tech stack-based implementation that you can set up more easily. It might just be that it needs some work to better connect the different repos, but I'm not working with it so I don't know the exact obstacles. I already heard that there could be funding opportunities.

But first of all, I think we should bring together all people working on related projects and check their status. I'm specifically thinking about discussing at least with:

• @clausmichele (EURAC) as contributor to openeo-odc and related repos
• @ValentinaHutter (or someone else) from EODC for their implementation
• @jonas-eberle (DLR) who seems to have an openEO implementation around the same tech stack
• @danielFlemstrom (RISE) who is developing also an implementation around ODC

Who would be interested in this?

cc @Open-EO/psc

[clausmichele]

Good point! I would be interested in this!
I think that currently the python xArray/Dask processing part is evolving nicely, but there is still a lack of a standardization in the back-end serving the openEO REST API. For instance, at Eurac we use the java based spring driver and I suppose at DLR as well (Prateek, one of the original developers, now work there). EODC is using a Python back-end and I do not know about RISE.

[aljacob]

I think this is definitely interesting and we should come up with some consolidated environment for quick deployment for this. This would definitely foster further uptake of openEO on the back-end provider side as well.

[jonas-eberle]

+1 for making it simpler and to consolidate the environments. I would also like to hear the pro's and con's about Java-based spring driver from EURAC and Python-based driver from EODC. We have also some experiences with using STAC API as a central data catalog used within xarray (Open Data Cube, stackstac).

[ValentinaHutter]

Sounds interesting, would be happy to join a call on it, if you plan to have one.

[christophreimer]

@m-mohr please add me to the discussion/meeting as well. I think we have to discuss the various components and/or software packages involved in an OpenDataCube / Pangeo backend for openEO

[m-mohr]

Yes, will do. Will send around a Doodle soon. Please note that this will not happen before April, maybe even only in May.

[jonas-eberle]

@m-mohr Hi Matthias, any update on this topic? Please also include @johntruckenbrodt - a new colleague at DLR who will work on xarray/Dask.

[m-mohr]

@jonas-eberle No, not yet. We need to find funding for this and then also find someone who has time to work on it. If anyone can contribute, please let us know.

[jonas-eberle]

Maybe it would be good to just start the discussion in a meeting - collecting some needs, requirements, ideas. This will also help us to estimate the amount of work.

[christophreimer]

Hi @jonas-eberle, we at EODC do have an Pangeo/ODC based backend around already. In addition we plan to release an "out-of-the-box" mini Pangeo/ODC backend by end July. My colleague @LukeWeidenwalker is currently working on that. Support for ODC is not foreseen to be included in this mini backend. If you have any questions, ideas or requirements, we are happy to discuss those with you.

[TomAugspurger]

Apologies for naive question, but can someone explain what we mean by "pangeo / ODC" stack here? Specifically, I'm wondering if pangeo = "JupyterHub + Dask on Kubernetes" in this context, or whether it's essentially just "xarray (plus rioxarray, etc.)".

[christophreimer]

@TomAugspurger we foresee to release a ready-to-use environment containing

• openEO processes implemented with xarray and
• the corresponding Dask packages
  packaged in Docker to run locally.

This is one of our building blocks we use at EODC to realize an openEO backend.
The EODC openEO backend makes use of the same openEO processes implemented in xarray, but allows for execution of those via Dask on k8s.

[m-mohr]

Not directly Pangeo/ODC/xarray related, but here's also documentation about VITO's back-end as local docker: https://github.com/Open-EO/openeo-geotrellis-kubernetes/blob/master/examples/run_local.md

[rabernat]

Nice to see this effort happening!

Pangeo provides docker images which it may be useful to build on here: https://github.com/pangeo-data/pangeo-docker-images