Bug: Fetching releases with similar names

[tsenovilla]

While working on #783, after including the Paseo chain specs in the pop up flow, I realized there's a subtle bug with how the releases are fetched when there are several latest releases at the same time. Namely, Paseo needed to release BridgeHub separately for a few releases, so we have https://github.com/paseo-network/runtimes/releases/tag/v2.0.1 and https://github.com/paseo-network/runtimes/releases/tag/v2.0.1-bh at the same time.

When calling pop up paseo, it sometimes picks one and sometimes picks the other one, leading to non-deterministic behavior as the command may launch the network or fail cause it doesn't find the needed artifacts.

PD: Paseo won't be affected by this issue soon, as the v2.0.2 release will bring back BridgeHub to the normal flow, but this issue might happen somewhere else in the future as it affects all pop sourcing.

[moliholy]

there are several latest releases at the same time

AFAIK there cannot be more than one release labeled as "latest" in a single repository 🤔

That's actually the case in your example, as https://github.com/paseo-network/runtimes/releases/tag/v2.0.1-bh is not the latest release.

[moliholy]

In order to fetch the latest release (labeled as such) we could simply use the latest_release() function. Not sure this is actually the case in the workflow you are referring to.

[tsenovilla]

Yeah you're right that a repo can have just a "latest" release. However somehow pop is sometimes fetching the v2.0.1-bh when using pop up. I didn't investigated this too much yet, but my guess is that it has to do with this line, as the tags v2.0.1 and v2.0.1-bh are prefixed in the same way and the tag comparator gets confused. I didn't touch the releases fetch flow so I guess it's not using the latest_release function you mentioned. But again, I didn't investigated this too much yet, just saw it was happening when I tried the functionality

[moliholy]

Will look into it. Thanks!

[AlexD10S]

I don't think is a pop-cli bug. The version v2.0.1-bh doesn't follow the usual convention, and it would be very difficult to design a proper solution for all possible variation, it the future you can add a -ph, -zh...

I think the -bh here is something temporal, a "special workaround", so I wouldn't make big changes because of this. Perhaps if we want to be sure it picks only one version we can simply ignore the suffix and only handle the conventional v2.0.1 format?

[moliholy]

@AlexD10S IMO the logic for fetching the "latest" version can be improved by using my suggestion. This way we yield an unique result.

[tsenovilla]

I agree with @moliholy that we should pick just the release marked as "latest".

However we should also be aware of this potential "issue", I agree with @AlexD10S that it's not a pop issue but something depending on a third party, but this doesn't mean we cannot find this in the future. Imagine you wanna run pop up paseo --parachains bridge-hub, that command won't never work as each spec is under different tags. Again, this is on Paseo's side cause it's not following the convention, but sometimes it's not possible to follow it during some releases (as it's the case here, BH needed different dependencies til 2.0.2). Someone using pop would be affected even if it's not pop's fault (or nobody's fault, maybe the third party had to act that way as Paseo did), but bad UX is on pop. So we might need to figure out if there's something we can do to mitigate this in a way we don't depend on what others do

[AlexD10S]

But I don't understand if is getting the 2.0.1 is already getting the latest no? At least is the one with the latest tag here: https://github.com/paseo-network/runtimes/releases

If a user specifically needs the 2.0.1-bh it can be specified manually with the flag:

-s, --system-parachain <SYSTEM_PARACHAIN>
          The version of the binary to be used for system parachains, as per the
          release tag (e.g. "stable2503"). Defaults to the relay chain version
          if not specified. See
          <https://github.com/paritytech/polkadot-sdk/releases> for more details

[moliholy]

@AlexD10S how do you define, exhaustively, "latest" tag? The one published most recently? The one labeled as "latest"? The first one when sorting alphabetically?

[tsenovilla]

But I don't understand if is getting the 2.0.1 is already getting the latest no? At least is the one with the latest tag here: https://github.com/paseo-network/runtimes/releases

It's not deterministically picking one, sometimes it picks 2.0.1, sometimes 2.0.1-bh. It's not picking the latest tag but a bunch of tags and then "ordering" them

If a user specifically needs the 2.0.1-bh it can be specified manually with the flag

The issue with specifically using the -s flag is that if you run pop up with two parachains (eg, coretime), it'll look for the other chain spec in the wrong tag (v2.0.1-bh in this case)

[AlexD10S]

@AlexD10S how do you define, exhaustively, "latest" tag? The one published most recently? The one labeled as "latest"? The first one when sorting alphabetically?

In the specific repo we are looking at the 2.0.1 has the latest tag: https://github.com/paseo-network/runtimes/releases/tag/v2.0.1 not the 2.0.1-bh. So even if we change the code to get the latest it will get this one no?.

In the conventional version tags, depends on the number: 3.0.0 > 2.0.1 and 2.0.2 > 2.0.1

[moliholy]

In the specific repo we are looking at the 2.0.1 has the latest tag: https://github.com/paseo-network/runtimes/releases/tag/v2.0.1 not the 2.0.1-bh

Indeed, but as @tsenovilla said, it is not deterministic. Sometimes it fetches the right one, and sometimes not. That's a bug.

[AlexD10S]

It's not deterministically picking one, sometimes it picks 2.0.1, sometimes 2.0.1-bh. It's not picking the latest tag but a bunch of tags and then "ordering" them

Ah ok! I understand now, then taking the latest as @moliholy suggested above will work

The issue with specifically using the -s flag is that if you run pop up with two parachains (eg, coretime), it'll look for the other chain spec in the wrong tag (v2.0.1-bh in this case)

But then the latest won't fix this issue either. We will need a mechanism to indicate one version per parachain you are launching no?

[tsenovilla]

But then the latest won't fix this issue either. We will need a mechanism to indicate one version per parachain you are launching no?

I don't know which is the best flow tbh, having such a mechanism would be probably overwhelming for the user.. That's what I meant in this comment. We should use latest as @moliholy said, that's for sure👍, but then maybe we should try to find out a way to mitigate these potential issues in our UX coming from what third parties do, using latest won't everything but it's a start as at least we always pick releases deterministically