[GitHub Request] move opensearch-k8s-operator functional-tests workflow to larger runners

[josedev-union]

What is the type of request?

No response

Details of the request

The functional-tests GitHub Actions job consistently fails because the default GitHub-hosted default runners cannot host an entire OpenSearch cluster plus the upgrade scenarios we run in CI. During provisioning/upgrade the job runs out of disk and is terminated. We need to move this workflow onto larger infrastructure so tests are reliable.

i.e. stand up self-hosted runners for the heavy job

• Provision a VM (or small pool) with spec bigger than standard Github runner size and register it as a runner with labels like self-hosted, linux, large.
• Limit the runner scope to this repo and dedicate it to functional tests only.
• Keep all other jobs (lint, unit tests, docs) on GitHub-hosted runners so we don’t waste dedicated resources on lightweight tasks.

Additional information to support your request

This is a blocker for merging open PRs in https://github.com/opensearch-project/opensearch-k8s-operator. CI job for operator functional tests always fails because of disk lack.

When does this request need to be completed?

No specific timeline, but the sooner the better.

Notes

Track the progress of your request here: Engineering Effectiveness Board (view).
Member of @opensearch-project/admin will take a look at the request soon.
Thanks!

[josedev-union]

@synhershko @prudhvigodithi here is the new issue

[synhershko]

@gaiksaya @prudhvigodithi can you assist with this?

[synhershko]

@prudhvigodithi @gaiksaya ping

[andrross]

Catch All Triage - 1 2 3

@cwperks mentioned there's another way to solve this if it is just an issue of disk space

[cwperks]

@josedev-union this problem has become more pronounced recently and you have a few options:

1. Remove some pre-installed software from the runners. Example: https://github.com/opensearch-project/security-dashboards-plugin/blob/main/.github/workflows/integration-test.yml#L100-L105

2. I believe @peterzhuamazon updated the project runners to remove pre-installed software automatically (Remove dotnet from container host to avoid space issues opensearch-build#5864)