OCPBUGS-67313: Fix race condition in external binary extraction

[stbenjam]

When multiple openshift-tests processes run concurrently and share the same cache directory, they can race on extracting and decompressing the same binaries. This commonly occurs when using shell pipelines like:

  openshift-tests run ... --dry-run | openshift-tests run -f -

Both processes start simultaneously, and without synchronization, one process may delete a .gz file while another is still trying to access it, resulting in errors like failed to delete original .gz file: no such file or directory.

Add flock-based file locking around the extraction process to ensure only one process extracts a given binary at a time. The lock is automatically released when the process exits, even if killed.

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[openshift-ci-robot]

@stbenjam: This pull request references Jira Issue OCPBUGS-67313, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

When multiple openshift-tests processes run concurrently and share the same cache directory, they can race on extracting and decompressing the same binaries. This commonly occurs when using shell pipelines like:

 openshift-tests run ... --dry-run | openshift-tests run -f -

Both processes start simultaneously, and without synchronization, one process may delete a .gz file while another is still trying to access it, resulting in errors like failed to delete original .gz file: no such file or directory.

Add flock-based file locking around the extraction process to ensure only one process extracts a given binary at a time. The lock is automatically released when the process exits, even if killed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@stbenjam: This pull request references Jira Issue OCPBUGS-67313. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

Details

In response to this:

When multiple openshift-tests processes run concurrently and share the same cache directory, they can race on extracting and decompressing the same binaries. This commonly occurs when using shell pipelines like:

 openshift-tests run ... --dry-run | openshift-tests run -f -

Both processes start simultaneously, and without synchronization, one process may delete a .gz file while another is still trying to access it, resulting in errors like failed to delete original .gz file: no such file or directory.

Add flock-based file locking around the extraction process to ensure only one process extracts a given binary at a time. The lock is automatically released when the process exits, even if killed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[liangxia]

/pipeline required

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[stbenjam]

/jira refresh

[openshift-ci-robot]

@stbenjam: This pull request references Jira Issue OCPBUGS-67313, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[stbenjam]

/jira refresh

[openshift-ci-robot]

@stbenjam: This pull request references Jira Issue OCPBUGS-67313, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[stbenjam]

/jira backport release-4.21,release-4.20

[openshift-ci-robot]

@stbenjam: The following backport issues have been created:

• OCPBUGS-70143 for branch release-4.21
• OCPBUGS-70144 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.21
/cherrypick release-4.20

Details

In response to this:

/jira backport release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-cherrypick-robot]

@openshift-ci-robot: once the present PR merges, I will cherry-pick it on top of release-4.20, release-4.21 in new PRs and assign them to you.

Details

In response to this:

@stbenjam: The following backport issues have been created:

• OCPBUGS-70143 for branch release-4.21
• OCPBUGS-70144 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[petr-muller]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [petr-muller,stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stbenjam]

/pipeline required

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[stbenjam]

/verified by CI

[openshift-ci-robot]

@stbenjam: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 71f668c and 2 for PR HEAD a46dd04 in total

[stbenjam]

/override ci/prow/okd-scos-images

[openshift-ci]

@stbenjam: Overrode contexts on behalf of stbenjam: ci/prow/okd-scos-images

Details

In response to this:

/override ci/prow/okd-scos-images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

@stbenjam: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@stbenjam: Jira Issue Verification Checks: Jira Issue OCPBUGS-67313
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-67313 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

When multiple openshift-tests processes run concurrently and share the same cache directory, they can race on extracting and decompressing the same binaries. This commonly occurs when using shell pipelines like:

 openshift-tests run ... --dry-run | openshift-tests run -f -

Both processes start simultaneously, and without synchronization, one process may delete a .gz file while another is still trying to access it, resulting in errors like failed to delete original .gz file: no such file or directory.

Add flock-based file locking around the extraction process to ensure only one process extracts a given binary at a time. The lock is automatically released when the process exits, even if killed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-cherrypick-robot]

@openshift-ci-robot: new pull request created: #30641

Details

In response to this:

@stbenjam: The following backport issues have been created:

• OCPBUGS-70143 for branch release-4.21
• OCPBUGS-70144 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@openshift-ci-robot: new pull request created: #30642

Details

In response to this:

@stbenjam: The following backport issues have been created:

• OCPBUGS-70143 for branch release-4.21
• OCPBUGS-70144 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-merge-robot]

Fix included in accepted release 4.22.0-0.nightly-2025-12-25-043524