release tests when sigterm/sigint is received

[ebarajas]

No description provided.

[yfang-figma]

Now looking at this again, I am not sure how much the additional release_command is going to help. We already shutdown! the queue, which should stop the polling loop, but apparently that did not happen (soon enough) for the the multibackend-permission job. And I think each CI worker would only hold one test at a time, so release_command most likely will only release just one test?
Anyway, it is worth trying.

[ebarajas]

Now looking at this again, I am not sure how much the additional release_command is going to help. We already shutdown! the queue, which should stop the polling loop, but apparently that did not happen (soon enough) for the the multibackend-permission job. And I think each CI worker would only hold one test at a time, so release_command most likely will only release just one test?

@yfang-figma I actually added this for the suite-based tests. If a worker dies after having reserved a large suite, it won't get picked back up until the dynamic timeout expires, and the timeout for those can be pretty large.

However I don't think we need this anymore, I added a step in CI that calls the release command if a worker is a retry.

[yfang-figma]

Now looking at this again, I am not sure how much the additional release_command is going to help. We already shutdown! the queue, which should stop the polling loop, but apparently that did not happen (soon enough) for the the multibackend-permission job. And I think each CI worker would only hold one test at a time, so release_command most likely will only release just one test?

@yfang-figma I actually added this for the suite-based tests. If a worker dies after having reserved a large suite, it won't get picked back up until the dynamic timeout expires, and the timeout for those can be pretty large.

However I don't think we need this anymore, I added a step in CI that calls the release command if a worker is a retry.

Oh, the first part of your reply makes sense. I am wondering for your new change, when can "worker is a retry" be detected and whether it is still faster to keep the release_command here?