Harvesting client improvements: configurable delay between GetRecord calls; a fix for a problem with long-running DataCite harvests

[landreev]

What this PR does / why we need it:

This is based on a patch that I made a while ago for another Dataverse instance. But it has come handy here at HDV and it may be of benefit to other instances out there.
The changes are quite straightforward.

From the accompanying release note:

A setting has been added for configuring sleep intervals between OAI calls for specific harvesting clients. Making it possible to harvest uninterrupted from servers enforcing rate limit policies. See the configuration guide for details. Additionally, this release fixes a problem with harvesting from DataCite OAI-PMH where initial, long-running harvests were failing on sets with large numbers of records.

Which issue(s) this PR closes:

• Closes Add a setting for configuring a rate limit in Harvesting Client (a limit on outgoing calls) #11473

Special notes for your reviewer:

Suggestions on how to test this:

Create a harvesting client and harvest something. For example, the controlTestSet from demo. Confirm that it's working without this new, optional setting.
Create the setting as described in the guide. If you configure an interval for the client you're testing, there will be an .info message in the log that the specified number of milliseconds will be used as the delay. If you specify it for a different client, confirm that the client you're testing is unaffected and the delay utilized is 0; i.e. no delay.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[coveralls]

coverage: 24.397% (-0.02%) from 24.414%
when pulling 5b87133 on 11473-harvesting-client-ratelimit
into f20e75a on develop.

[landreev]

Un-drafting this thing.

[landreev]

(will sync w/ develop shortly)

[landreev]

Synced the branch with develop just now. There were no merge conflicts to resolve however, contrary to what GitHub was saying. 🤔

[landreev]

@stevenwinship Thanks for adding the test - looks good.

[landreev]

I just killed and deleted the latest Jenkins built, since it was triggered by a change in the release note.
The last Jenkins job that ran did in fact succeed.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:11473-harvesting-client-ratelimit

ghcr.io/gdcc/configbaker:11473-harvesting-client-ratelimit

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[sekmiller]

If you add the setting via the api with a bad json file you still get a success message "Setting :HarvestingClientCallRateLimit added", but when you run the harvest you get the "Disabling all harvesting client delay intervals completely until fixed!" . in the log. Is there any way that we can note this on setting create or put the error in the UI on Harvest instead of just in the log?

[landreev]

Hmm. I based this implementation on, or copied it from, how Oliver implemented a similar map setting for tabular ingest size limits. So yes, it is possible to create it with junk values that are only validated when used; similarly to :TabularIngestSizeLimit.

I couldn't think of a completely trivial/cheap way to quickly add validation on create. So I'd like to petition to let it slip. On the rationale that this one is an even more exotic feature than the tab. ingest limits; and something only sysadmins - advanced users, by definition - will have to tinker with; very rarely, if ever.

Do we ever validate any other settings on create though?

If you add the setting via the api with a bad json file you still get a success message "Setting :HarvestingClientCallRateLimit added", but when you run the harvest you get the "Disabling all harvesting client delay intervals completely until fixed!" . in the log. Is there any way that we can note this on setting create or put the error in the UI on Harvest instead of just in the log?