BTF type information is missing since recent kernel update 6.6.95, making eBPF tools such as Tetragon crash on start-up

[blu3r4y]

What Happened?

Symptom

I was installing CNCF Tetragon, a very popular eBPF-based security and observability tool in Minikube v1.37.0 on Windows and it failed with the following message:

level=error msg="Failed to execute tetragon" error="tetragon, aborting kernel autodiscovery failed: kernel version \"6.6.95\" BTF search failed kernel is not included in supported list. Please check Tetragon requirements documentation, then use --btf option to specify BTF path and/or '--kernel' to specify kernel version"
level=info msg="BTF discovery: default kernel btf file does not exist" btf-file=/sys/kernel/btf/vmlinux
level=info msg="BTF discovery: candidate btf file does not exist" btf-file=/var/lib/tetragon/metadata/vmlinux-6.6.95
level=info msg="BTF discovery: candidate btf file does not exist" btf-file=/var/lib/tetragon/btf

This worked with Minikube v1.36.0 on Windows.

Potential root cause

I did some digging in #20995 and found that flags CONFIG_DEBUG_INFO=y and CONFIG_DEBUG_INFO_BTF=y were removed: https://github.com/kubernetes/minikube/pull/20995/files#diff-fa5e1b16477a9509c25c6d3c289baac93c8456b6a58afb7f087d732842a7a542L519-L520

Was this on purpose? It would be very valuable to have BTF type information, especially in Kubernetes, where many operators use eBPF technology (and BTF type information).

Footnote

I also came across #19886 and #8556 which mention the tutorial (https://minikube.sigs.k8s.io/docs/tutorials/ebpf_tools_in_minikube/) that seems very outdated; e.g., minikube ssh -- docker ps cannot connect to the Docker domain.

Attach the log file

log.txt

Operating System

Windows

Driver

Hyper-V

[nirs]

I did some digging in #20995 and found that flags CONFIG_DEBUG_INFO=y and CONFIG_DEBUG_INFO_BTF=y were removed: https://github.com/kubernetes/minikube/pull/20995/files#diff-fa5e1b16477a9509c25c6d3c289baac93c8456b6a58afb7f087d732842a7a542L519-L520

Was this on purpose? It would be very valuable to have BTF type information, especially in Kubernetes, where many operators use eBPF technology (and BTF type information).

This was a fallout of cleaning up the buildroot configuration and upgrading the kernel.

Do you want to send a PR enabling them again?

Note that we don't add options manually. You need to run:

make linux-menuconfig-x86_64
make linux-menuconfig-aarch64

and enable the relevant options. This make the next upgrade easier.

See https://minikube.sigs.k8s.io/docs/contrib/building/iso/ for more info on building minikube iso.

[blu3r4y]

Yes, I am working on a PR right now. However, my first freshly built kernel is still missing BTF type information, although I re-added the options. I will check again in a few days what I am missing here.

[nirs]

Yes, I am working on a PR right now. However, my first freshly built kernel is still missing BTF type information, although I re-added the options.

Sometimes options names change and what worked with older kernel does not work with a new kernel. Enabling the options using menuconfig helps to get the right configs are added.

[nirs]

Required configuration:
https://tetragon.io/docs/installation/faq/#what-are-the-linux-kernel-configuration-options-needed-to-run-tetragon

[blu3r4y]

Thanks for the hint with make linux-menuconfig-x86_64; I was missing CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT which also needs to be enabled for CONFIG_DEBUG_INFO_BTF to work. Opened PR #21800

[nirs]

/assign @blu3r4y

[nirs]

/kind bug