[FEATURE] Template for ML resource cleanup

[martin-gaievski]

Is your feature request related to a problem?

Currently, Flow Framework has several pre-configured templates for deploying remote models and creating corresponding connectors: https://docs.opensearch.org/latest/automating-configurations/workflow-templates/#supported-workflow-templates. However, there is
no similar template that allows cleanup of ML resources. This is needed for scenarios involving one-time autonomous actions that need to create resources during a setup step and tear them down after the action is completed.

What solution would you like?

Template that can roll back resources create by "model deployment" templates:

• undeploy model
• delete model
• delete model group
• delete connector

What alternatives have you considered?

Perform resource cleanup via individual API calls. This lowers user adoption because such operations require deeper knowledge of OpenSearch internals and are error-prone.

Do you have any additional context?

This new functionality would be helpful for search relevance experiment scenarios, when a user does not have a connector to an LLM but wants to use features like LLM-generated judgments: https://docs.opensearch.org/latest/search-plugins/search-relevance/
using-search-relevance-workbench

[dbwiddis]

Hey @martin-gaievski, we do have the deprovision API: https://docs.opensearch.org/latest/automating-configurations/api/deprovision-workflow/

It iterates through whatever resources have been deployed for a given workflow and undeploys/deletes them.

And the individual workflow steps are available if you want to have a more finely grained control, although that's a bit more of an unusual setup and/or workaround.

Can you describe your need/use-case a bit more specifically?

[martin-gaievski]

In my scenario I need to delete resources associated with remote model: model group, model, connector. I think that's very similar to what is created by one of flow existing templates for model deployment, e.g. cohere_embedding_model_deploy.

The API for de-provisioning that you've mentioned, should work in theory. I've run smoke test with resources created by that template for deploying Cohere embedding model.

I'm seeing partially correct behavior: some of the resources are deleted (model and connector), but not all of them (model group stays). Can you please check my steps, maybe something is missing/wrong, I bumped into known limitation

1. Provision connector to Cohere model:

POST /_plugins/_flow_framework/workflow?use_case=cohere_embedding_model_deploy&provision=true
{
    "create_connector.credential.key" : "<MY_API_KEY>"
}

I can see that workflow is created:

{
  "workflow_id": "fqa3r5sBH57jLXRJ5A6b",
  "state": "COMPLETED",
  "resources_created": [
    {
      "resource_type": "connector_id",
      "resource_id": "f6a3r5sBH57jLXRJ5A7c",
      "workflow_step_name": "create_connector",
      "workflow_step_id": "create_connector"
    },
    {
      "resource_type": "model_id",
      "resource_id": "gqa3r5sBH57jLXRJ5Q4q",
      "workflow_step_name": "register_remote_model",
      "workflow_step_id": "register_model"
    },
    {
      "resource_type": "model_id",
      "resource_id": "gqa3r5sBH57jLXRJ5Q4q",
      "workflow_step_name": "deploy_model",
      "workflow_step_id": "deploy_model"
    }
  ]
}

2. Verify that resources are actually created
   model group

{
                "_index": ".plugins-ml-model-group",
                "_id": "gKa3r5sBH57jLXRJ5Q4B",
                "_version": 2,
                "_seq_no": 6,
                "_primary_term": 1,
                "_score": 1.0,
                "_source": {
                    "created_time": 1768179623169,
                    "access": "public",
                    "latest_version": 1,
                    "last_updated_time": 1768179623195,
                    "name": "Cohere english embed model",
                    "description": "cohere-embedding-model"
                }
            }

model

{
                "_index": ".plugins-ml-model",
                "_id": "gqa3r5sBH57jLXRJ5Q4q",
                "_version": 3,
                "_seq_no": 58,
                "_primary_term": 1,
                "_score": 1.0,
                "_source": {
                    "last_deployed_time": 1768179623292,
                    "model_version": "1",
                    "created_time": 1768179623210,
                    "deploy_to_all_nodes": true,
                    "is_hidden": false,
                    "description": "cohere-embedding-model",
                    "model_state": "DEPLOYED",
                    "planning_worker_node_count": 3,
                    "auto_redeploy_retry_times": 0,
                    "last_updated_time": 1768179623293,
                    "name": "Cohere english embed model",
                    "connector_id": "f6a3r5sBH57jLXRJ5A7c",
                    "current_worker_node_count": 3,
                    "model_group_id": "gKa3r5sBH57jLXRJ5Q4B",
                    "planning_worker_nodes": [
                        "0Ak48oRrQ_STPv2kY59AGQ",
                        "ynHhRhjrS9ytIdksUH7EfQ",
                        "32nL12oVSLW4OOfgXXwo0w"
                    ],
                    "algorithm": "REMOTE"
                }
            }

connector

{
                "_index": ".plugins-ml-connector",
                "_id": "f6a3r5sBH57jLXRJ5A7c",
                "_version": 1,
                "_seq_no": 2,
                "_primary_term": 1,
                "_score": 1.0,
                "_source": {
                    "created_time": 1768179623132,
                    "protocol": "http",
                    "last_updated_time": 1768179623132,
                    "name": "cohere-embedding-connector",
                    "description": "The connector to Cohere's public embed API",
                    "version": "1",
                    "parameters": {
                        "endpoint": "${{create_connector.endpoint}}",
                        "truncate": "END",
                        "input_type": "search_document",
                        "model": "embed-english-v3.0"
                    },
                    "actions": [
                        {
                            "headers": {
                                "Authorization": "Bearer ${credential.key}",
                                "Request-Source": "unspecified:opensearch"
                            },
                            "post_process_function": "connector.post_process.cohere.embedding",
                            "pre_process_function": "connector.pre_process.cohere.embedding",
                            "method": "POST",
                            "request_body": "{ \"texts\": ${parameters.texts}, \"truncate\": \"${parameters.truncate}\", \"model\": \"${parameters.model}\", \"input_type\": \"${parameters.input_type}\" }",
                            "action_type": "PREDICT",
                            "url": "https://api.cohere.ai/v1/embed"
                        }
                    ]
                }
            }

3. Deprovision resources created by the workflow on step 1

POST /_plugins/_flow_framework/workflow/fqa3r5sBH57jLXRJ5A6b/_deprovision

4. Check workflow status

GET /_plugins/_flow_framework/workflow/fqa3r5sBH57jLXRJ5A6b/_status

{
  "workflow_id": "fqa3r5sBH57jLXRJ5A6b",
  "state": "NOT_STARTED"
}

5. Check resources. Model and connector are deleted, but model_group still exists

GET {{base_url}}/_plugins/_ml/model_groups/_search
{
  "query": {
    "match_all": {}
  }
}

            {
                "_index": ".plugins-ml-model-group",
                "_id": "gKa3r5sBH57jLXRJ5Q4B",
                "_version": 2,
                "_seq_no": 6,
                "_primary_term": 1,
                "_score": 1.0,
                "_source": {
                    "created_time": 1768179623169,
                    "access": "public",
                    "latest_version": 1,
                    "last_updated_time": 1768179623195,
                    "name": "Cohere english embed model",
                    "description": "cohere-embedding-model"
                }
            }

[dbwiddis]

Thanks for the details, @martin-gaievski ! I'm still a bit confused, though!

• The output for step 1 only has the connector (created) and model (registered and deployed). These are exactly the resources that will be deprovisioned with the deprovision API.
• I do see in step 2 that there is a model group created that's attached to the model id. If the model group were created as part of the workflow template, it would have been included in the created resources. But in this case, the model group is automatically created.
• I was under the understanding that model groups automatically created with/attached to a single model which created them were automatically deleted when the associated model is deleted. This seems to either not be happening, or being enforced by some cleanup that may take time.

We DO have workflow steps to delete things like model groups, and you could create a template to do that cleanup, but you'd need to know the model group ID to call them. And that would require some of the APIs you are calling to search, by which points direct deletion is much better than trying to create a template to do the deletion.

So it boils down to this follow up question: is the model group the only thing remaining? And does it persist more than the ML cron interval (I think 10 or 20 seconds) the refresh interval? If not, that's expected. If so, this may be either be a bug in ML Commons we need to follow up on, or a misunderstanding of how models and automatically-created model groups interact.

[dbwiddis]

I just realized another likely situation.

The model group is auto deleted with the model but deleted documents can still show up in a search until the next refresh of the index.

So try a get-by-id to see if it’s actually gone, or refresh the model group index before searching.

[martin-gaievski]

thanks @dbwiddis, I've done additional checks and looks like refresh of the ml index should be out of the picture. I've done both force index refresh, still model_group is present, same with get-by-id query, it returns model_group document.
Will try to figure out how model_group is created from ml-commons perspective, and if there is an option to auto-delete it if it has been auto-created together with connector or with the model.

[dbwiddis]

Will try to figure out how model_group is created from ml-commons perspective, and if there is an option to auto-delete it if it has been auto-created together with connector or with the model.

From the docs: https://docs.opensearch.org/latest/ml-commons-plugin/api/model-group-apis/delete-model-group/

When you delete the last model version in a model group, that model group is automatically deleted from the index.

That used to happen...

Could you check in a few versions to help bisect the regression? I have a suspicion this slipped in in 3.4 but don't see anything obvious yet.

[dbwiddis]

Interesting: @rbhavna submitted this PR to "fix" this feature back in October 2023:

• fix model group auto-deletion when last version is deleted ml-commons#1444

But the diff shows it actually removed that functionality?

        client.delete(deleteRequest, new ActionListener<DeleteResponse>() {
            @Override
            public void onResponse(DeleteResponse deleteResponse) {
-                if (isLastModelOfGroup) {
-                    deleteModelGroup(modelGroupId);
-                }
                deleteModelChunks(modelId, deleteResponse, actionListener);
            }

But I swear it's worked since then, so maybe the functionality is elsewhere...

[martin-gaievski]

It looks to me that the problem is on ml-commons side. I just delete all resources manually via their API, and still model group remains untouched. Logged an issue for ml-commons repo: opensearch-project/ml-commons#4536

[dbwiddis]

Yeah, I see that as well. Actions we could take on FF side:

• explicitly create model groups in our templates so they show up as resources that will get deprovisioned (not ideal)
• gather information when a model is created about any auto-created model group and if it should be deleted, and add it as a resource
  • We already do something like this for the auto-deploy case as shown above for the remote model
  • ideally that information is in the ml client response, otherwise we'd need to do a search to find it (not ideal)

Better action is to try to address this on ML side to have some "auto created" flag/name/unique identifier that allows "auto delete"