Integration Test: TestDownloadOnly is flaky and should be refactored #21805
Description
Looking in the test is does not disable preloads, and it checks if reload exists after starting minikube. If the preload tarball exists it does check for binaries. If preload tarball does not exit, it fallback to checking the binaries.
This test has few problems:
- it is too complicated
- it does not test the same case in all runs, depending on external state. If the preload tarball exist in the external system it test one case, and if not another case.
preloadExists := false
t.Run("preload-exists", func(t *testing.T) {
// skip for none, as none driver does not have preload feature.
if NoneDriver() {
t.Skip("None driver does not have preload")
}
// Driver does not matter here, since the only exception is none driver,
// which cannot occur here.
if !download.PreloadExists(v, containerRuntime, "docker", true) {
t.Skip("No preload image")
}
// Just make sure the tarball path exists
if _, err := os.Stat(download.TarballPath(v, containerRuntime)); err != nil {
t.Errorf("failed to verify preloaded tarball file exists: %v", err)
}
preloadExists = true
})
t.Run("cached-images", func(t *testing.T) {
// skip verify for cache images if --driver=none
if NoneDriver() {
t.Skip("None driver has no cache")
}
if preloadExists {
t.Skip("Preload exists, images won't be cached")
}
imgs, err := images.Kubeadm("", v)
if err != nil {
t.Errorf("failed to get kubeadm images for %v: %+v", v, err)
}
for _, img := range imgs {
pathToImage := []string{localpath.MiniPath(), "cache", "images", runtime.GOARCH}
img = strings.Replace(img, ":", "_", 1) // for example kube-scheduler:v1.15.2 --> kube-scheduler_v1.15.2
imagePath := strings.Split(img, "/") // changes "gcr.io/k8s-minikube/storage-provisioner_v5" into ["gcr.io", "k8s-minikube", "storage-provisioner_v5"] to match cache folder structure
pathToImage = append(pathToImage, imagePath...)
fp := filepath.Join(pathToImage...)
_, err := os.Stat(fp)
if err != nil {
t.Errorf("expected image file exist at %q but got error: %v", fp, err)
}
}
})
We need is to test all cases in all runs:
- Preload does not exist, falling back to downloaded images - change preload url/name so it cannot be downloaded from the system publishing the preloads. The best way to test it is to add a --preload-url flag. We can make this flag hidden if we don't think it is useful or users and we need it only for testing.
- Preload exists, no fallback - must use a preload that always exists. Using latest release is a good bet. Since preloads are important minikube feature, we can assume that we always have a preload for the latest release. If it does not work it is likely to be fixed.
- Preload disabled (--preload=false), fallback to downloading the binaries.
Note that we also have preload_test.go, which tests --preload=false. But it does not test downloaded binaries. It seem to be focused on something else, not sure how it is related to preloads.
Metadata
Metadata
Assignees
Labels
No labels