Inquiry Regarding Time Intervals Between Consecutive Updates in Routinator

[Meadows1254465703]

I am writing to share some observations regarding the time intervals between consecutive validation runs in Routinator, based on our recent analysis of its log outputs.

To investigate the update intervals, we calculated the time differences between the start times of consecutive validation runs by analyzing log entries marked with "Starting a validation run." This allowed us to derive the distribution of these intervals.

We conducted this analysis using Routinator v0.14.2 with default parameters (including the documented refresh value of 10 minutes, which should delay the next update until 10 minutes after the previous run completes). Notably, the observed intervals exhibited a bimodal distribution across two independent machines:

• Machine 1:

• Machine 2:

While the second peak in the distribution aligns with the expected 10-minute refresh interval described in the documentation, the first peak suggests unexpected behavior. Further examination of the logs revealed frequent occurrences of the entry:
[INFO] Next validation run scheduled in 0 seconds
This implies that certain validation runs initiated immediately after the previous run concluded, deviating from the intended 10-minute delay.

Could you kindly help us understand the following:
1.What scenarios might trigger a validation run to restart with a 0-second delay despite the default refresh setting?
2.Are there additional configurations or external factors that could influence this behavior?

We sincerely appreciate your expertise and time in addressing these questions. Please let us know if further details or logs would assist your investigation.

[partim]

Hi! Thank you for doing this research and reaching out! We have had reports of this before but somehow haven’t gotten around to properly figuring out how and why this happens. As you can imagine, this is quite tricky to get to the ground of.

One question that came up discussing this here just now was whether these 0-second waits happen in sequence or if typically the next wait is normal. Would you perhaps be willing to provide us with the raw data you did your calculations on? That would perhaps allow us to discover other patterns as well. If you don’t want to publish the data, you can also send it to us privately at rpki-team@nlnetlabs.nl.

[Meadows1254465703]

Hi! Thank you for doing this research and reaching out! We have had reports of this before but somehow haven’t gotten around to properly figuring out how and why this happens. As you can imagine, this is quite tricky to get to the ground of.

One question that came up discussing this here just now was whether these 0-second waits happen in sequence or if typically the next wait is normal. Would you perhaps be willing to provide us with the raw data you did your calculations on? That would perhaps allow us to discover other patterns as well. If you don’t want to publish the data, you can also send it to us privately at rpki-team@nlnetlabs.nl.

Thank you for your prompt follow-up! I completely understand the challenges of debugging such nuanced behavior, and I’m happy to share the raw data to assist your investigation.

I attempted to upload the files directly to GitHub, but encountered issues with the platform’s attachment system. Instead, I’ve sent the two datasets ​via email​ to rpki-team@nlnetlabs.nl as you suggested.

[Koenvh1]

We think we have a solution for this in #1027

[ties]

@Koenvh1 does this change fetch + revalidate, or re-validate only?

I would ratelimit re-validations because of expired objects to prevent degraded behaviour (this was in RIPE NCC RPKI Validator 3). But as a repository operator, it would be inconvenient if all routinator instances started re-validating every [default min-refresh] if a repository had it's expiring objects spread out through time this way.

Optionally, a RTR speaker can handle the per-VRP/ASPA/BGPsec expiration at the RTR level if the timestamp is in the JSON. I think that is in the stayrtr code.

[eKrajnak]

@ties I think, every time you expire any ROA, you need to do new fetch (to look if ROA covering the same prefix was created/updated). Adding expiration timestamp to JSON can cause inconsistency.

[ties]

@ties I think, every time you expire any ROA, you need to do new fetch (to look if ROA covering the same prefix was created/updated). Adding expiration timestamp to JSON can cause inconsistency.

The update interval is relatively short (10 minutes default). New objects are published far (hours) before expiration.

Fetching on expiry can cause a multiplication of the number of fetches. And how do you handle the object after it expired? Do you wait for the fetch, while you know it is no longer valid?

[eKrajnak]

Yes, default update interval in Routinator is 10min. But in practice, Tier1 operators (any many others) perform update in much longer interval (maybe hours).

And yes, new objects are USUALLY published hours before. But what if not? Publishing new object very shortly before expiration can cause inconsistency I've mentioned.

Rare cases, but we need to take them into account.

[ties]

And yes, new objects are USUALLY published hours before. But what if not? Publishing new object very shortly before expiration can cause inconsistency I've mentioned.

Do you have data for that?

Rare cases, but we need to take them into account.

It also is just "CA intent " to use expiry at set times, but not make the time identical among objects.

When we had this implemented in the ripe ncc validator, we did see cases where objects expired very close to eachother. Effectively a single misconfigured CA can multiply the load on repositories by the ratio between default refresh and default min-refresh.

I think that the situation was strictly worse before min-refresh though

[partim]

@Koenvh1 does this change fetch + revalidate, or re-validate only?

We currently only do fetch + revalidate (with an exception for the first run after starting in server mode in the next release).