Arbitrary CWL DAG deployment

[jpl-btlunsfo]

Purpose

• Automatically generate a basic CWL-executing DAG
• Deployable using the executionUnit parameters of an OGC request

Proposed Changes

• Adding cwl template file
• Changing deploy_process method to create file if executionunit is present

Issues

• Automatically generate a DAG that can execute an arbitrary CWL workflow unity-sps#92

Testing

• tested output of templating script in personal airflow deployment, successfully ran echo cwl task

- Successfully deployed multiple DAGs through the OGC api

[jpl-btlunsfo]

Successfully deployed templated DAG through a modified OGC endpoint!

Going forward, the request payload schema will be just a little bit different (still abiding by https://docs.ogc.org/DRAFTS/20-044.html)

[jpl-btlunsfo]

Example request (for a cwl dag):

creating

URL (POST to): http://k8s-sps-ogcproce-64159ac933-118635354.us-west-2.elb.amazonaws.com:5001/processes
Body:

{
    "executionUnit":{
        "type": "application/cwl",
        "href": "https://raw.githubusercontent.com/unity-sds/unity-sps-workflows/main/demos/echo_message.cwl"
    },
    "processDescription": {
        "id": "test_cwl_dag1",
        "title": "testing cwl dag1",
        "description": "placeholder description",
        "version": "1.0.2",
        "jobControlOptions": [
            "async-execute",
            "sync-execute"
        ],
        "inputs": [],
        "outputs": [],
        "links": [
            {
            "href": "https://processing.example.org/oapi-p/processes/EchoProcess/execution",
            "rel": "http://www.opengis.net/def/rel/ogc/1.0/execute",
            "title": "Execute endpoint"
            }
        ]
    }
}

deleting

URL (send a DELETE to): http://k8s-sps-ogcproce-64159ac933-118635354.us-west-2.elb.amazonaws.com:5001/processes/test_cwl_dag2
(no body necessary)