[RFE] Option to disable permanent networking setup in the bastion host

[josecastillolema]

Today jetlag permanently configures the networking on the bastion host (using nmcli) so as the config would persist across reboots on the bastion host.
While this is ideal for the typical manual user in the labs, this is not ideal from the CI perspective (and it's becoming increasingly complex to manage).
What we are suggesting here is a new var (permanent_network_bastion_setup), that when set to false prevents this configuration to be persistent. That way, at the begining at a Prow CI run, we can just systemctl network restart and make sure we have a clean and consistent starting point.

[josecastillolema]

cc @rsevilla87 @mcornea

[akrzos]

I don't think we need to solve this with yet another var. We need to decide if the primary focus should be convenience for typical deployments or CI consumers. I still think the primary usability should be focused on lab users since CI is scripted and should understood what is occurring, if that is not the case how can we even trust what the CI is doing if we don't know what is occurring? What is too difficult about removing known configuration of network interfaces that the CI use case can not handle?

[josecastillolema]

I still think the primary usability should be focused on lab users

CI is increasingly become a first citizen of jetlag, and this will only naturally evolve with time as we got more use cases, self-scheduling, etc. Plus, we could leave the default of this new var the way is more convenient for manual users, I dont see why we can cover both scenarios.

how can we even trust what the CI is doing

They way to trust a CI is having ephemeral stateless reproducible builds. If we cannot trust the state of the environment because there are left-overs from previous runs this becomes an issue.

What is too difficult about removing known configuration of network interfaces that the CI use case can not handle?

How do you propose to do that?
Looking at the nmcli setup of one bastion host, how do you know which ones where configured as part of jetlag? An option here would be to add connection.user-data metadata to the interfaces jetlag configures so we could filter then. The option I am proposing just adds a var and a --temporary flag to the connections when it makes sense, happy to move forward either way.

[akrzos]

I think this issue is mainly because there is a lack of understanding of how the network is setup. I would suggest you to review the network diagram in the docs so you can understand what network interfaces are in play here. For the bastion, there is only one interface in use. We actually can not make it any less interfaces if we wanted to. Can the prow job not clear this interface at the conclusion of the job? If the job fails, can it not just clear every interface not related to the lab interface of the bastion? I think there are many methods to solve this without adding another variable. Also a self-scheduling job should never need this since it is receiving clean machines from the start every single time. Are the self-scheduling machines showing up with left over configuration? If so I would say this is a lab related issue and completely outside of the scope of jetlag.

[jtaleric]

Good discussion here.

While most Jetlag deploys through CI are for "dedicated" CI runs, there are occasions where folks (myself) take over the BM for different use cases -- mainly customer or research. With CI, we expect the state to be static and never changing -- however with BM, there is the chance morons like myself or CI hiccups leave the environment in a non-ideal state.

Instead of having a var added why don't we just add the user-data to mark the interface as a Jetlag interface?

Alternatively (ugly, but an alternative). in CI we store the output from Jetlag in /tmp, we could find the latest ansible-setup-bastion-XXXX -- we could do something like this

[root@m42-h01-000-r760 tmp]# grep "Display NetworkManager" $(ls -p -t ansible-setup-bastion-* | head -1) -A 4 | grep msg | awk '{print $2}'
"ens1f0"

However, with the user-data it will be more obvious and should be transparent with no need for a new var.