Same server -> different interface naming

[josecastillolema]

We need a way to deal with this situation where the same HW type can have different interface names, an example:

• fxy-h04-xyz-r630 -> enp4s0f0
• fxy-h10-xyz-r630-> enp3s0f0

The current approach relies on the lab.yml file and will fail when we get assigned r630 servers with enp4s0f0:

  scalelab:
    r630:
    - enp3s0f0
    - eno1
    - eno2
    - eno3
    - eno4

Some ideas to consider:

• Use the mac address provided by the ocpinventory.json file instead? Can we rely on the order of the interfaces for this?
• Check if the interface exists on the server before the deployment using the QUADS API?

$ curl -s https://quads2.example/api/v3/hosts/fxy-h04-xyz-r630.example.com | jq '.interfaces[] | select(.name == "em1")'
{
  "bios_id": null,
  "host_id": xyz,
  "id": xyz,
  "mac_address": "xx:xx:xx:xx:xx:xx",
  "maintenance": false,
  "name": "em1",
  "pxe_boot": false,
  "speed": 10,
  "switch_ip": "x.y.z.w",
  "switch_port": "xe-0/x/x:x",
  "vendor": ""
}

$ curl -s https://quads2.example.com/api/v3/hosts/f24-h04-000-r630.example.com | jq '.interfaces[] | select(.name == "em1dsd")'

• Something similar to what jetlag does, maintaining few exception for certain racks:

      r640:
        pub_nic: eno1np0
        prov_nic: ens1f1
        bm_nic: ens2f0np0
        bm_nic_2: eno2np1
        prov_nic_f18: ens3f1

[josecastillolema]

cc @mcornea @grafuls

[josecastillolema]

cc @smalleni how this was handled in jetski?

[akrzos]

Have we spoke to the lab folks to find out if perhaps the cables just got swapped between interfaces or if this is a separate network card, perhaps it was plugged into the incorrect PCI port?

[josecastillolema]

What I am being told is that the devops team cannot guarantee interface naming and we should handle this from the automation standpoint 😢

[jtaleric]

This reminds me of the OpenStack deployment issues we would always run into. We ended up obfuscating the interface name with nic1, nic2 etc...

When we query from Quads, all we get is em1,em2,em3,etc. Which also doesn't really help from the OS interface name... so the best solution is to use the MAC address. Also the fact that not all BM nodes will be uniform, even if it is the same model (r630).

However, I am not following how we update the mapping with the MACs @josecastillolema ?

[sadsfae]

@josecastillolema @jtaleric the differences here are expected because that specific host: f24-h04-000-r630 likely has its NIC in a physically different PCI slot than the rest and this is what interface naming conventions via RHEL and systemd-udevd are based on.

That host also has a different disk path too:

This one uses for example: disk/by-path/pci-0000:03:00.0-scsi-0:2:0:0
The rest of the R630 use: disk/by-path/pci-0000:02:00.0-scsi-0:2:0:0

This is a one-off.

You can rely on device naming to be identical so long as the version of RHEL that your OCP images are based on don't decide to change it in a systemd-udev update unless all the models are not identical. The Scale Lab default right now is RHEL9.4, it's not unheard of for this to change even between minor u-releases. This bites us quite often too and we have to go and update the docs.

The R630 have a few one-off specs, one of them has 8x disks for example but AFAICT only f24-h04-000-r630 could have a different interface name for one of the NICS due to slot placement.

PS: Please remove internal FQDN from public github comments and use .example.com instead - infosec gets mad at us otherwise.

[sadsfae]

What I am being told is that the devops team cannot guarantee interface naming and we should handle this from the automation standpoint 😢

What we mean by this is that we can't control or even know about when systemd-udev updates decide to change interface naming conventions if it's not the lab default OS. Some level of checking they match what you expect is likely in order here to be sure because this does change. It happened twice during EL8 and at least once during EL9 lifecycle and we had to scramble and update our docs and Foreman templates to adjust.

[sadsfae]

We need a way to deal with this situation where the same HW type can have different interface names, an example:

* fxy-h04-xyz-r630 -> enp4s0f0

* fxy-h10-xyz-r630-> enp3s0f0

The current approach relies on the lab.yml file and will fail when we get assigned r630 servers with enp4s0f0:

  scalelab:
    r630:
    - enp3s0f0
    - eno1
    - eno2
    - eno3
    - eno4

Some ideas to consider:

* Use the mac address provided by the **ocpinventory.json** file instead? Can we rely on the order of the interfaces for this?

* Check if the interface exists on the server before the deployment using the QUADS API?

$ curl -s https://quads2.rdu2.scalelab.redhat.com/api/v3/hosts/fxy-h04-xyz-r630.rdu2.scalelab.redhat.com | jq '.interfaces[] | select(.name == "em1")'
{
  "bios_id": null,
  "host_id": xyz,
  "id": xyz,
  "mac_address": "xx:xx:xx:xx:xx:xx",
  "maintenance": false,
  "name": "em1",
  "pxe_boot": false,
  "speed": 10,
  "switch_ip": "x.y.z.w",
  "switch_port": "xe-0/x/x:x",
  "vendor": ""
}

$ curl -s https://quads2.rdu2.scalelab.redhat.com/api/v3/hosts/f24-h04-000-r630.rdu2.scalelab.redhat.com | jq '.interfaces[] | select(.name == "em1dsd")'

* Something similar to what jetlag does, maintaining few exception for certain racks:

      r640:
        pub_nic: eno1np0
        prov_nic: ens1f1
        bm_nic: ens2f0np0
        bm_nic_2: eno2np1
        prov_nic_f18: ens3f1

@josecastillolema we try very hard in Scale Lab to have uniformity across models, for the most part we achieve this but sometimes we inherit sets of systems other groups have ordered which are out-of-spec. Other times things are ordered and thrown over the fence afterwards and we have to rely on the QUADS metadata model to sort it all out.

As a suggestion to your solution above, on the Jetlag process side what if you had a YAML override that used the short hostname for the times when they aren't uniform? You could identify this with Ansible facts and auto-generate these templates to match what you were assigned - this way if what we call an interface on the lab default differs from a newer systemd-udevd and RHEL minor release update that's newer than what we run you have the correct interface name as our docs would not show that.

If you deconstruct the interface naming convention you can implement logic to distinguish what is what (onboard versus PCI, port number etc).

@grafuls suggests we could look at adding something like an os_name value to the interface model in QUADS that the API can query but the problem there is keeping it up to date when a newer RHEL update / systemd-udevd update OCP or CoreOS might be rebased upon decides to change it again in an update newer than what the lab default is.

[josecastillolema]

Hi @sadsfae !
I totally agree that we need a solution to deal with this kind of situations.
If I understood correctly, there is the only one R630 host with a different naming for the interfaces?
If this is the case, could we please consider removing this one from the self-scheduling pool while we work on a permanent solution? cc @grafuls

[sadsfae]

Hi @sadsfae ! I totally agree that we need a solution to deal with this kind of situations. If I understood correctly, there is the only one R630 host with a different naming for the interfaces? If this is the case, could we please consider removing this one from the self-scheduling pool while we work on a permanent solution? cc @grafuls

Hey @josecastillolema I already removed it from self-scheduling after submitting that last github comment the other day, no worries. I am not 100% certain but I believe it's the only one that's different. If you come across others let us know.

I am nearly positive no other system otherwise will have this issue across other models for when more powerful systems can make their way back to the SSM pool.

[akrzos]

@josecastillolema @jtaleric the differences here are expected because that specific host: f24-h04-000-r630 likely has its NIC in a physically different PCI slot than the rest and this is what interface naming conventions via RHEL and systemd-udevd are based on.

Would it be possible to schedule maintenance for this machine in which we move its NIC card to the same PCI slot as the other r630s? I would assume this should help close the gap on this machine's difference from the others.

On another note I recall and found in my notes where at least one r630 (f27-h34-000-r630) had a different public lab nic than what we original observed and what was documented in the wiki. At the time I worked on the machine (back in June) its public nic was enp129s0f0 as opposed to enp3s0f0. I am also unsure if the original owner of the cloud at the time may have done something or changed operating systems on this system such that it showed that nic name instead.

[sadsfae]

@josecastillolema @jtaleric the differences here are expected because that specific host: f24-h04-000-r630 likely has its NIC in a physically different PCI slot than the rest and this is what interface naming conventions via RHEL and systemd-udevd are based on.

Would it be possible to schedule maintenance for this machine in which we move its NIC card to the same PCI slot as the other r630s? I would assume this should help close the gap on this machine's difference from the others.

On another note I recall and found in my notes where at least one r630 (f27-h34-000-r630) had a different public lab nic than what we original observed and what was documented in the wiki. At the time I worked on the machine (back in June) its public nic was enp129s0f0 as opposed to enp3s0f0. I am also unsure if the original owner of the cloud at the time may have done something or changed operating systems on this system such that it showed that nic name instead.

Sure we are happy to take a look, let's tackle that in a PERFINFRA JIRA and @kambiz-aghaiepour can take a poke when he's in the datacenter next time. We could ask IT datacenter team to physically look for us too.