add --merge-json cli flag to support bktec and pants

[jasonwbarnett]

This feature is related to adding pants support to test-engine-client (i.e. buildkite/test-engine-client#323)

[nprizal]

Hi @jasonwbarnett. Thank's for opening this PR. I tested this locally and the file merging is working 🎉. However, I noticed that the .lock file is not being deleted after the process, is this normal? Also do we need create the lock when --merge-json is disabled?

[jasonwbarnett]

@nprizal Thanks for testing this and for the feedback!

You're absolutely right — I've updated the code so that we only acquire a lock when --merge-json is used.

As for the .lock file being left behind — that's expected behavior when using the filelock package. It doesn’t delete the lock file after releasing it, by design. The presence of the file isn’t what enforces the lock; the lock is handled either via OS-level mechanisms or by writing a PID into the file. Deleting the file can actually introduce race conditions, so it's safest to leave it.

I also made a second change: we're no longer silently rescuing JSON parse errors. If there's malformed JSON in the file, it'll now raise an exception as it should.

Let me know if anything else comes up!

[nprizal]

@jasonwbarnett Thanks for explaining how the Filelock package works. The changes look good to me. However, there are some lint issues and failed tests on the CI. Could you please have a look?

[jasonwbarnett]

🎉 🎉 🎉 🎉

[jasonwbarnett]

@nprizal Once this PR is merged and a new version is published to pypi.org I can wrap up buildkite/test-engine-client#323 since it requires the newly published package for those tests to pass.

Specific test which is failing due to not having this published: https://buildkite.com/buildkite/test-engine-client/builds/1511/steps/canvas?jid=019756b8-20cd-4086-a4a8-8cff717330ac#019756b8-20cd-4086-a4a8-8cff717330ac/266-289

[nprizal]

@jasonwbarnett Yes, I've checked your other PR. I'll let you know when this change is published to pypi. Thanks again for your contribution 😃

[nprizal]

@jasonwbarnett The new version has been published to pypi 🚢🐍 https://pypi.org/project/buildkite-test-collector/1.0.4/