-
Notifications
You must be signed in to change notification settings - Fork 12
Adding docs for EOAP_CWL on CDSE. #842
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
025a6fd
Adding docs for EOAP_CWL on CDSE. https://github.com/eu-cdse/openeo-c…
EmileSonneveld 1cad8f3
Fix links. Tweaked docs a bit. https://github.com/eu-cdse/openeo-cdse…
EmileSonneveld 5a52c6c
Add contact information. Fix some writing. https://github.com/eu-cdse…
EmileSonneveld 17acbc2
Respond to MR
EmileSonneveld ed7d067
Respond to MR
EmileSonneveld 51c7da3
Respond to MR
EmileSonneveld 7d7905f
clearify.
EmileSonneveld File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| --- | ||
| title: "EOAP CWL" | ||
| execute: | ||
| echo: false | ||
| jupyter: python3 | ||
| aliases: | ||
| - /EOAP-CWL.html | ||
| --- | ||
|
|
||
| With openEO, it is now possible to run CWL ([Common Workflow Language](https://www.commonwl.org/)) in the `run_udf` process. | ||
| Workflows will be executed using Calrissian on Kubernetes. | ||
| First read the general CWl documentation from `openeo-geopyspark-driver`: [udf-eoap-cwl.md](https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/udf-eoap-cwl.md) | ||
| On the CDSE backend, there are some extra features, which are described on this page. | ||
|
|
||
|
|
||
| ## S3 access | ||
|
|
||
| CWL workflows running on this backend will receive short-lived S3 credentials with read-only access to the `eodata` bucket on CDSE. | ||
| Those credentials will be available in the following environment variables: | ||
|
|
||
| - `AWS_ENDPOINT_URL_S3` | ||
| - `AWS_ACCESS_KEY_ID` | ||
| - `AWS_SECRET_ACCESS_KEY` | ||
|
|
||
| They only work inside the cluster environment, and are only temporarily valid. | ||
| You can use these instead of your own credentials. This way, your Docker images can remain public. | ||
|
|
||
| ## Docker images | ||
|
|
||
| CWL allows to use docker images to run code. For example: | ||
|
|
||
| ```yaml | ||
| requirements: | ||
| - class: DockerRequirement | ||
| dockerPull: ghcr.io/cloudinsar/openeo_insar:20260219T1446 | ||
| ``` | ||
|
|
||
| Only whitelisted Docker images can be used in the cluster. Contact us | ||
| through [support](https://helpcenter.dataspace.copernicus.eu/hc/en-gb/requests/new) if you have custom images that needs | ||
| to be whitelisted. (You might need to create an account) | ||
| As of February 2026, only Docker images that can be pulled without credentials are used. | ||
|
|
||
| ## Memory limits | ||
|
|
||
| Increasing requested memory increases credit usage. | ||
| The maximum amount of memory available is deployment specific, but would be around 20Gb. | ||
| If your job gets stuck without being processed, consider lowering the requested memory. | ||
|
|
||
| ## Debugging locally | ||
|
|
||
| To test your CWL workflow locally before running it on the cluster, | ||
| you can use [cwltool](https://pypi.org/project/cwltool/) locally. | ||
| You might need to provide your own S3 credentials. You can request them | ||
| here: https://documentation.dataspace.copernicus.eu/APIs/S3.html | ||
|
|
||
| ```bash | ||
| cwltool \ | ||
| --tmpdir-prefix=$HOME/tmp/ \ | ||
| --force-docker-pull \ | ||
| --no-read-only \ | ||
| --parallel \ | ||
| --preserve-environment=AWS_ENDPOINT_URL_S3 \ | ||
| --preserve-environment=AWS_ACCESS_KEY_ID \ | ||
| --preserve-environment=AWS_SECRET_ACCESS_KEY \ | ||
| example_workflow.cwl example_parameters.json | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems a lot of mentioned options are not essential or even risky to use unless you know what you are doing. I would just have the essentials here and suggest some more advanced options seperately
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed 2 unneeded parameters. Annoyingly,
--tmpdir-prefixand--no-read-onlyare needed to be able to run for me.