From 025a6fdd3e41160ca27fe37ef4e1f4903f9edc38 Mon Sep 17 00:00:00 2001 From: Emile Sonneveld Date: Thu, 19 Feb 2026 16:43:08 +0100 Subject: [PATCH 1/7] Adding docs for EOAP_CWL on CDSE. https://github.com/eu-cdse/openeo-cdse-infra/issues/838 --- APIs/openEO/EOAP-CWL.qmd | 54 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 APIs/openEO/EOAP-CWL.qmd diff --git a/APIs/openEO/EOAP-CWL.qmd b/APIs/openEO/EOAP-CWL.qmd new file mode 100644 index 0000000000..5bff6c897c --- /dev/null +++ b/APIs/openEO/EOAP-CWL.qmd @@ -0,0 +1,54 @@ +--- +title: "EOAP CWL" +execute: + echo: false +jupyter: python3 +aliases: + - /EOAP-CWL.html +--- + +With openEO, it is now possible to run CWL workflows. On the CDSE backend, this has some extra features, which are described on this page. +Backend agnostic information can be found here: https://github.com/Open-EO/openeo-geopyspark-driver/pull/1548/changes (TODO: Update URL once merged). + + +## S3 access + +CWL code running on this backend will receive short-lived S3 credentials to access the `eodata` bucket. +Those credentials can be accessed through the following environment variables: + +- `AWS_ENDPOINT_URL_S3` +- `AWS_ACCESS_KEY_ID` +- `AWS_SECRET_ACCESS_KEY` + +They won't work outside the cluster, and are only temporarily valid. +You can use these instead of your own credentials. This way, your Docker images can remain public. + +## Docker images + +Only whitelisted Docker images can be used in the cluster. Contact us through the forum or support to get your image prefix whitelisted. + + +## Memory limits + +At the time of writing (February 2026), the memory limit is set to 15GB. + +## Debugging locally + +You might want to test your CWL workflow locally before running it on the cluster. +To do this, you can use [cwltool](https://pypi.org/project/cwltool/) locally. +You might need to provide your own S3 credentials. You can request them +here: https://documentation.dataspace.copernicus.eu/APIs/S3.html + +```bash +cwltool \ + --tmpdir-prefix=$HOME/tmp/ \ + --force-docker-pull \ + --leave-container \ + --leave-tmpdir \ + --no-read-only \ + --parallel \ + --preserve-environment=AWS_ENDPOINT_URL_S3 \ + --preserve-environment=AWS_ACCESS_KEY_ID \ + --preserve-environment=AWS_SECRET_ACCESS_KEY \ + example_workflow.cwl example_parameters.json +``` From 1cad8f3364518c275bcfb966dc620852ac545aae Mon Sep 17 00:00:00 2001 From: Emile Sonneveld Date: Fri, 20 Feb 2026 09:08:18 +0100 Subject: [PATCH 2/7] Fix links. Tweaked docs a bit. https://github.com/eu-cdse/openeo-cdse-infra/issues/838 --- APIs/openEO/EOAP-CWL.qmd | 5 +++-- _quarto.yml | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/APIs/openEO/EOAP-CWL.qmd b/APIs/openEO/EOAP-CWL.qmd index 5bff6c897c..e7a0403a47 100644 --- a/APIs/openEO/EOAP-CWL.qmd +++ b/APIs/openEO/EOAP-CWL.qmd @@ -8,12 +8,12 @@ aliases: --- With openEO, it is now possible to run CWL workflows. On the CDSE backend, this has some extra features, which are described on this page. -Backend agnostic information can be found here: https://github.com/Open-EO/openeo-geopyspark-driver/pull/1548/changes (TODO: Update URL once merged). +Backend agnostic information can be found here: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) ## S3 access -CWL code running on this backend will receive short-lived S3 credentials to access the `eodata` bucket. +CWL code running on this backend will receive short-lived S3 credentials with read-only access to the `eodata` bucket. Those credentials can be accessed through the following environment variables: - `AWS_ENDPOINT_URL_S3` @@ -26,6 +26,7 @@ You can use these instead of your own credentials. This way, your Docker images ## Docker images Only whitelisted Docker images can be used in the cluster. Contact us through the forum or support to get your image prefix whitelisted. +At the moment of writing, only public Docker images are used. ## Memory limits diff --git a/_quarto.yml b/_quarto.yml index bc8084f8bd..05052376be 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -138,6 +138,8 @@ website: text: Data Collections - href: "APIs/openEO/Processes.qmd" text: openEO Processes + - href: "APIs/openEO/EOAP-CWL.qmd" + text: EOAP CWL - href: "APIs/openEO/File_formats.qmd" text: File Formats - section: Authentication From 5a52c6c27c79a3eee2299cb2bad0b09eecec1459 Mon Sep 17 00:00:00 2001 From: Emile Sonneveld Date: Mon, 23 Feb 2026 09:56:27 +0100 Subject: [PATCH 3/7] Add contact information. Fix some writing. https://github.com/eu-cdse/openeo-cdse-infra/issues/838 --- APIs/openEO/EOAP-CWL.qmd | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/APIs/openEO/EOAP-CWL.qmd b/APIs/openEO/EOAP-CWL.qmd index e7a0403a47..21a5146577 100644 --- a/APIs/openEO/EOAP-CWL.qmd +++ b/APIs/openEO/EOAP-CWL.qmd @@ -7,36 +7,37 @@ aliases: - /EOAP-CWL.html --- -With openEO, it is now possible to run CWL workflows. On the CDSE backend, this has some extra features, which are described on this page. -Backend agnostic information can be found here: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) +With openEO, it is now possible to run CWL ([Common Workflow Language](https://www.commonwl.org/)) in the `run_udf` process. This is executed on Calrissian +On the CDSE backend, this has some extra features, which are described on this page. +More information can be found here: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) ## S3 access -CWL code running on this backend will receive short-lived S3 credentials with read-only access to the `eodata` bucket. +CWL workflows running on this backend will receive short-lived S3 credentials with read-only access to the `eodata` bucket. Those credentials can be accessed through the following environment variables: - `AWS_ENDPOINT_URL_S3` - `AWS_ACCESS_KEY_ID` - `AWS_SECRET_ACCESS_KEY` -They won't work outside the cluster, and are only temporarily valid. +They only work inside the cluster environment, and are only temporarily valid. You can use these instead of your own credentials. This way, your Docker images can remain public. ## Docker images -Only whitelisted Docker images can be used in the cluster. Contact us through the forum or support to get your image prefix whitelisted. -At the moment of writing, only public Docker images are used. +Only whitelisted Docker images can be used in the cluster. Contact us through [support](https://helpcenter.dataspace.copernicus.eu/hc/en-gb/requests/new) (You might need to create an account) +As of February 2026, only public Docker images are used. ## Memory limits -At the time of writing (February 2026), the memory limit is set to 15GB. +As of February 2026, the memory limit is set to 15GB. ## Debugging locally -You might want to test your CWL workflow locally before running it on the cluster. -To do this, you can use [cwltool](https://pypi.org/project/cwltool/) locally. +To test your CWL workflow locally before running it on the cluster, +you can use [cwltool](https://pypi.org/project/cwltool/) locally. You might need to provide your own S3 credentials. You can request them here: https://documentation.dataspace.copernicus.eu/APIs/S3.html From 17acbc2545245d67a4b7a98dcd12896fa5f02e6d Mon Sep 17 00:00:00 2001 From: Emile Sonneveld Date: Mon, 23 Feb 2026 15:06:06 +0100 Subject: [PATCH 4/7] Respond to MR --- APIs/openEO/EOAP-CWL.qmd | 17 +++++++++++------ _quarto.yml | 4 ++-- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/APIs/openEO/EOAP-CWL.qmd b/APIs/openEO/EOAP-CWL.qmd index 21a5146577..5d62de1570 100644 --- a/APIs/openEO/EOAP-CWL.qmd +++ b/APIs/openEO/EOAP-CWL.qmd @@ -8,14 +8,14 @@ aliases: --- With openEO, it is now possible to run CWL ([Common Workflow Language](https://www.commonwl.org/)) in the `run_udf` process. This is executed on Calrissian -On the CDSE backend, this has some extra features, which are described on this page. -More information can be found here: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) +First read the general CWl documentation from `openeo-geopyspark-driver`: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) +On the CDSE backend, there are some extra features, which are described on this page. ## S3 access CWL workflows running on this backend will receive short-lived S3 credentials with read-only access to the `eodata` bucket. -Those credentials can be accessed through the following environment variables: +Those credentials will be available in the following environment variables: - `AWS_ENDPOINT_URL_S3` - `AWS_ACCESS_KEY_ID` @@ -26,8 +26,15 @@ You can use these instead of your own credentials. This way, your Docker images ## Docker images +CWL allows to use docker images to run code. For example: + +```yaml +requirements: + - class: DockerRequirement + dockerPull: ghcr.io/cloudinsar/openeo_insar:20260219T1446 +``` Only whitelisted Docker images can be used in the cluster. Contact us through [support](https://helpcenter.dataspace.copernicus.eu/hc/en-gb/requests/new) (You might need to create an account) -As of February 2026, only public Docker images are used. +As of February 2026, only Docker images that can be requested without credentials are used. ## Memory limits @@ -45,8 +52,6 @@ here: https://documentation.dataspace.copernicus.eu/APIs/S3.html cwltool \ --tmpdir-prefix=$HOME/tmp/ \ --force-docker-pull \ - --leave-container \ - --leave-tmpdir \ --no-read-only \ --parallel \ --preserve-environment=AWS_ENDPOINT_URL_S3 \ diff --git a/_quarto.yml b/_quarto.yml index 05052376be..a4252b59d7 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -138,8 +138,6 @@ website: text: Data Collections - href: "APIs/openEO/Processes.qmd" text: openEO Processes - - href: "APIs/openEO/EOAP-CWL.qmd" - text: EOAP CWL - href: "APIs/openEO/File_formats.qmd" text: File Formats - section: Authentication @@ -181,6 +179,8 @@ website: target: "_blank" - href: "APIs/openEO/fair.qmd" text: FAIR & open science + - href: "APIs/openEO/EOAP-CWL.qmd" + text: EOAP CWL - section: "Sentinel Hub" href: "APIs/SentinelHub.qmd" contents: From ed7d0672ed0e8aeade3311b6b677f5c92d9ac1af Mon Sep 17 00:00:00 2001 From: Emile Sonneveld Date: Mon, 23 Feb 2026 15:16:04 +0100 Subject: [PATCH 5/7] Respond to MR --- APIs/openEO/EOAP-CWL.qmd | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/APIs/openEO/EOAP-CWL.qmd b/APIs/openEO/EOAP-CWL.qmd index 5d62de1570..e92a33a8b7 100644 --- a/APIs/openEO/EOAP-CWL.qmd +++ b/APIs/openEO/EOAP-CWL.qmd @@ -7,7 +7,8 @@ aliases: - /EOAP-CWL.html --- -With openEO, it is now possible to run CWL ([Common Workflow Language](https://www.commonwl.org/)) in the `run_udf` process. This is executed on Calrissian +With openEO, it is now possible to run CWL ([Common Workflow Language](https://www.commonwl.org/)) in the `run_udf` process. +Workflows will be executed using calrissian on kubernetes. First read the general CWl documentation from `openeo-geopyspark-driver`: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) On the CDSE backend, there are some extra features, which are described on this page. @@ -34,7 +35,7 @@ requirements: dockerPull: ghcr.io/cloudinsar/openeo_insar:20260219T1446 ``` Only whitelisted Docker images can be used in the cluster. Contact us through [support](https://helpcenter.dataspace.copernicus.eu/hc/en-gb/requests/new) (You might need to create an account) -As of February 2026, only Docker images that can be requested without credentials are used. +As of February 2026, only Docker images that can be pulled without credentials are used. ## Memory limits From 51c7da386e6cabfaeb336d5c29e9455e76d93736 Mon Sep 17 00:00:00 2001 From: Emile Sonneveld Date: Thu, 26 Feb 2026 15:07:32 +0100 Subject: [PATCH 6/7] Respond to MR --- APIs/openEO/EOAP-CWL.qmd | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/APIs/openEO/EOAP-CWL.qmd b/APIs/openEO/EOAP-CWL.qmd index e92a33a8b7..b5f310c998 100644 --- a/APIs/openEO/EOAP-CWL.qmd +++ b/APIs/openEO/EOAP-CWL.qmd @@ -37,10 +37,11 @@ requirements: Only whitelisted Docker images can be used in the cluster. Contact us through [support](https://helpcenter.dataspace.copernicus.eu/hc/en-gb/requests/new) (You might need to create an account) As of February 2026, only Docker images that can be pulled without credentials are used. - ## Memory limits -As of February 2026, the memory limit is set to 15GB. +Increasing requested memory increases credit usage. +The max amount of memory available is deployment specific, but would be around 20Gb. +If your job gets stuck without being processed, consider lowering the requested memory. ## Debugging locally From 7d7905f9e335f827daa2a9e098fc6c755094853e Mon Sep 17 00:00:00 2001 From: Emile Sonneveld Date: Thu, 26 Feb 2026 15:27:03 +0100 Subject: [PATCH 7/7] clearify. --- APIs/openEO/EOAP-CWL.qmd | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/APIs/openEO/EOAP-CWL.qmd b/APIs/openEO/EOAP-CWL.qmd index b5f310c998..1b723d704b 100644 --- a/APIs/openEO/EOAP-CWL.qmd +++ b/APIs/openEO/EOAP-CWL.qmd @@ -8,14 +8,14 @@ aliases: --- With openEO, it is now possible to run CWL ([Common Workflow Language](https://www.commonwl.org/)) in the `run_udf` process. -Workflows will be executed using calrissian on kubernetes. +Workflows will be executed using Calrissian on Kubernetes. First read the general CWl documentation from `openeo-geopyspark-driver`: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) On the CDSE backend, there are some extra features, which are described on this page. ## S3 access -CWL workflows running on this backend will receive short-lived S3 credentials with read-only access to the `eodata` bucket. +CWL workflows running on this backend will receive short-lived S3 credentials with read-only access to the `eodata` bucket on CDSE. Those credentials will be available in the following environment variables: - `AWS_ENDPOINT_URL_S3` @@ -34,13 +34,16 @@ requirements: - class: DockerRequirement dockerPull: ghcr.io/cloudinsar/openeo_insar:20260219T1446 ``` -Only whitelisted Docker images can be used in the cluster. Contact us through [support](https://helpcenter.dataspace.copernicus.eu/hc/en-gb/requests/new) (You might need to create an account) + +Only whitelisted Docker images can be used in the cluster. Contact us +through [support](https://helpcenter.dataspace.copernicus.eu/hc/en-gb/requests/new) if you have custom images that needs +to be whitelisted. (You might need to create an account) As of February 2026, only Docker images that can be pulled without credentials are used. ## Memory limits Increasing requested memory increases credit usage. -The max amount of memory available is deployment specific, but would be around 20Gb. +The maximum amount of memory available is deployment specific, but would be around 20Gb. If your job gets stuck without being processed, consider lowering the requested memory. ## Debugging locally