Occasionally boatload will say cleanup phase completed however it did not

[akrzos]

It seems sometimes boatload running kube-burner will say that cleanup phase completed however it had not actually completed and some namespaces as left behind, which are then potentially caught in the next iteration run when running many boatload tests.

2021-11-04 01:23:02,942 : INFO : Output : time="2021-11-03 20:23:02" level=info msg="Setting log level to info"
2021-11-04 01:23:02,942 : INFO : Output : time="2021-11-03 20:23:02" level=info msg="🔥 Starting kube-burner with UUID 3f748950-1e55-4b8b-8ff5-68f372c02198"
2021-11-04 01:23:02,942 : INFO : Output : time="2021-11-03 20:23:02" level=info msg="📁 Creating indexer: elastic"
2021-11-04 01:23:03,265 : INFO : Output : time="2021-11-03 20:23:03" level=info msg="📈 Creating measurement factory"
2021-11-04 01:23:03,265 : INFO : Output : time="2021-11-03 20:23:03" level=info msg="Registered measurement: podLatency"
2021-11-04 01:23:03,271 : INFO : Output : time="2021-11-03 20:23:03" level=info msg="Preparing delete job: cleanup-boatload"
2021-11-04 01:23:03,271 : INFO : Output : time="2021-11-03 20:23:03" level=info msg="Job cleanup-boatload: Delete Namespace with selector kube-burner-job=boatload"
2021-11-04 01:23:03,271 : INFO : Output : time="2021-11-03 20:23:03" level=info msg="Triggering job: cleanup-boatload"
2021-11-04 01:23:03,367 : INFO : Output : time="2021-11-03 20:23:03" level=info msg="Found 418 namespaces with selector kube-burner-job=boatload, removing them"
2021-11-04 09:04:35,437 : INFO : Output : time="2021-11-04 04:04:35" level=info msg="Job cleanup-boatload took 27692.17 seconds"
2021-11-04 09:04:35,437 : INFO : Output : time="2021-11-04 04:04:35" level=info msg="Indexing metadata information for job: cleanup-boatload"
2021-11-04 09:04:36,005 : INFO : Output : time="2021-11-04 04:04:36" level=info msg="Indexing [1] documents in boatload-default"
2021-11-04 09:04:36,067 : INFO : Output : time="2021-11-04 04:04:36" level=info msg="Indexed [1] documents in 61ms in boatload-default"
2021-11-04 09:04:36,067 : INFO : Output : time="2021-11-04 04:04:36" level=info msg="Finished execution with UUID: 3f748950-1e55-4b8b-8ff5-68f372c02198"
2021-11-04 09:04:36,067 : INFO : Output : time="2021-11-04 04:04:36" level=info msg="👋 Exiting kube-burner"

Left over terminating namespaces:

# oc get ns | grep boat
boatload-218                                       Terminating   11h
boatload-308                                       Terminating   11h
boatload-372                                       Terminating   11h
boatload-381                                       Terminating   11h
# oc describe ns | grep kube-burner-uuid
              kube-burner-uuid=3f748950-1e55-4b8b-8ff5-68f372c02198
              kube-burner-uuid=3f748950-1e55-4b8b-8ff5-68f372c02198
              kube-burner-uuid=3f748950-1e55-4b8b-8ff5-68f372c02198
              kube-burner-uuid=3f748950-1e55-4b8b-8ff5-68f372c02198