Rate limit

[stbischof]

I try to adjust my client that i can work with the x-ratelimit .

the docs say:
https://docs.clearlydefined.io/docs/get-involved/using-data

All other endpoints are max 2K requests per minute.

after running 5 requests like:

curl -X GET "https://api.clearlydefined.io/definitions/npm/npmjs/%40yarnpkg/lockfile/1.1.13" -H "accept: */*" -v

I expected to have
x-ratelimit-remaining: 1995

bit is was

x-ratelimit-limit: 2000 
x-ratelimit-remaining: 1893
x-ratelimit-reset: 1742743739

this way it is hard to calc when and how often i can do requests, since the number remaining seems to decreases for each request differently

if i use the

post on definitions i always get

x-ratelimit-limit: 0 
x-ratelimit-remaining: 0
x-ratelimit-reset: 174xxxxxxx

so i never know before when i get an 429.

what is the best way to respect your api restrictions but can do efficient requests

[stbischof]

any updates?

[qtomlinson]

The pull request to update the rate limit libraries has been merged. The development deployment, accessible at https://dev-api.clearlydefined.io/, is now available for testing. Your feedback is greatly appreciated!

[stbischof]

GET and POST works as expected,
Reset seems also to work.

Would love to see it in production

[waynebeaton]

@qtomlinson our tool only calls ClearlyDefined a small number of times (usually < ~10). But we're still getting HTTP 429 periodically.

When we created the tool, our intention was that it would be used from the developer's desktop via CLI, but some clever folks figured out how to integrate the tool into their CI builds on GitHub.

My hypothesis that that we're getting HTTP 429s because many multiple projects are running the tool in GitHub actions at the same time and the ClearlyDefined server sees them all coming from the same place. I'm further thinking/assuming that this is made worse by many other GitHub projects running other tools/scans on GitHub infrastructure that also hammer on ClearlyDefined.

We didn't think too hard about rate limiting because the number of calls that we make are small. We can certainly implement this (we have a pull request waiting to be merged), but I'm thinking that this doesn't solve the problem that I've described. It would only -- maybe -- defer it for a bit.

We didn't think too hard about authenticating for the same reason... But I'm wondering if authenticating might solve the problem... does the rate limiting infrastructure grant a particular authenticated identity a number of calls, or would it put multiple authenticated entities all coming from the same infrastructure into a single bucket?

[waynebeaton]

Over the past several weeks, I've been getting an HTTP 429 on my very first use of the API.

[ryanbrandenburg]

@stbischof and @waynebeaton can you check if you're still experiencing this? We deployed #1368 yesterday, it should fix this scenario.

And a big thanks to both of you, your clear descriptions of the behavior got me thinking about what ended up being the solution.

[waynebeaton]

Thanks for doing this work. I haven't had any trouble today. I'll poke at it some more.

[sschuberth]

BTW, we've been experiencing similar issues in the ORT project over the last weeks, and today we're getting a lot of SocketTimeoutExceptions.