Verifier Bug on Newer Kernels

[airend]

Starting with kernel 6.17, I get this unsettling REG INVARIANTS VIOLATION:

Nov 17 09:56:10 LEGION kernel: ------------[ cut here ]------------
Nov 17 09:56:10 LEGION kernel: verifier bug: REG INVARIANTS VIOLATION (false_reg2): range bounds violation u64=[0xd, 0xc] s64=[0xd, 0xc] u32=[0xd, 0xc] s32=[0xd, 0xc] var_off=(0xc, 0x0)
Nov 17 09:56:10 LEGION kernel: WARNING: CPU: 13 PID: 3609264 at kernel/bpf/verifier.c:2721 reg_bounds_sanity_check+0x1b1/0x1c0
Nov 17 09:56:10 LEGION kernel: Modules linked in: mimic(OE) nf_tables qmi_wwan cdc_wdm uinput uhid rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device ccm algif_aead des3_ede_x86_64 des_generic libdes algif_skcipher cmac md4 bnep algif_hash af_alg btusb btbcm btintel btrtl btmtk bluetooth vfat fat snd_sof_amd_acp70 snd_sof_amd_acp63 snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_xtensa_dsp snd_sof_pci snd_ctl_led snd_sof snd_hda_codec_alc269 intel_rapl_msr amd_atl snd_sof_utils snd_hda_scodec_component intel_rapl_common snd_hda_codec_realtek_lib snd_soc_core iwlmvm snd_hda_codec_nvhdmi snd_hda_codec_generic snd_compress snd_hda_codec_hdmi ptp snd_pci_ps kvm_amd snd_hda_intel pps_core joydev snd_soc_acpi_amd_match mousedev mac80211 snd_hda_codec snd_rpl_pci_acp6x kvm snd_acp_pci libarc4 snd_hda_core snd_amd_acpi_mach snd_acp_legacy_common r8169 snd_intel_dspcfg snd_pci_acp6x ee1004 snd_hwdep snd_pci_acp5x ucsi_acpi realtek irqbypass sp5100_tco typec_ucsi mdio_devres snd_rn_pci_acp3x
Nov 17 09:56:10 LEGION kernel:  polyval_clmulni snd_pcm iwlwifi roles ghash_clmulni_intel snd_acp_config ideapad_laptop i2c_piix4 libphy snd_timer aesni_intel typec hid_multitouch snd_soc_acpi cm32181 rapl sparse_keymap wmi_bmof cfg80211 pcspkr platform_profile wdat_wdt snd ccp zenpower(OE) i2c_smbus mdio_bus snd_pci_acp3x soundcore thunderbolt rfkill industrialio i2c_hid_acpi i2c_hid mac_hid tcp_bbr sch_cake pkcs8_key_parser i2c_dev crypto_user acpi_call(OE) loop dm_mod nfnetlink ip_tables x_tables ntfs3 rndis_host cdc_ether usbnet mii wireguard libcurve25519 ip6_udp_tunnel udp_tunnel lz4 lz4_compress nvidia_uvm(O) nvidia_drm(O) nvidia_modeset(O) nvme nvme_core nvme_keyring nvme_auth nvidia(O) hkdf serio_raw amdgpu drm_panel_backlight_quirks drm_buddy drm_suballoc_helper video wmi drm_exec i2c_algo_bit drm_display_helper cec gpu_sched amdxcp drm_ttm_helper ttm
Nov 17 09:56:10 LEGION kernel: CPU: 13 UID: 955 PID: 3609264 Comm: mimic Tainted: G           OE       6.18.0-rc5-10-cachyos-rc #1 PREEMPT(full)  cda57a506fb05520506a00a8b84e362fb1db4c7e
Nov 17 09:56:10 LEGION kernel: Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 17 09:56:10 LEGION kernel: Hardware name: LENOVO 82JQ/LNVNB161216, BIOS GKCN65WW 01/16/2024
Nov 17 09:56:10 LEGION kernel: RIP: 0010:reg_bounds_sanity_check+0x1b1/0x1c0
Nov 17 09:56:10 LEGION kernel: Code: 54 44 8b 6b 48 8b 6b 4c 49 89 c1 ff 73 20 ff 73 18 55 41 55 41 53 41 52 ff 73 30 e8 59 7c be ff 4c 89 e1 4c 89 fa 48 83 c4 38 <0f> 0b 4c 8b 43 38 4c 8b 4b 40 e9 9d fe ff ff 90 90 90 90 90 90 90
Nov 17 09:56:10 LEGION kernel: RSP: 0018:ffffcc534f8876d0 EFLAGS: 00010292
Nov 17 09:56:10 LEGION kernel: RAX: 3bcd315e4985f700 RBX: ffff8b231b68b310 RCX: ffffffffa17b5726
Nov 17 09:56:10 LEGION kernel: RDX: ffffffffa1786785 RSI: 0000000000000027 RDI: ffff8b2b6f15cd08
Nov 17 09:56:10 LEGION kernel: RBP: 000000000000000c R08: 000000000000018a R09: ffffffffa22598d0
Nov 17 09:56:10 LEGION kernel: R10: 000000000000049e R11: 00000000fffff18a R12: ffffffffa17b5726
Nov 17 09:56:10 LEGION kernel: R13: 000000000000000d R14: ffff8b2a66d18000 R15: ffffffffa1786785
Nov 17 09:56:10 LEGION kernel: FS:  00007fa501afe740(0000) GS:ffff8b2bcc174000(0000) knlGS:0000000000000000
Nov 17 09:56:10 LEGION kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 17 09:56:10 LEGION kernel: CR2: 00007f9b52df1000 CR3: 0000000118cd3000 CR4: 0000000000f50ef0
Nov 17 09:56:10 LEGION kernel: PKRU: 55555554
Nov 17 09:56:10 LEGION kernel: Call Trace:
Nov 17 09:56:10 LEGION kernel:  <TASK>
Nov 17 09:56:10 LEGION kernel:  check_cond_jmp_op+0x8e6/0xbd0
Nov 17 09:56:10 LEGION kernel:  do_check+0x279d/0x4230
Nov 17 09:56:10 LEGION kernel:  do_check_common+0x42c/0xbb0
Nov 17 09:56:10 LEGION kernel:  bpf_check+0x5834/0x5b60
Nov 17 09:56:10 LEGION kernel:  ? mod_memcg_state+0x9f/0x1e0
Nov 17 09:56:10 LEGION kernel:  ? pcpu_alloc_noprof+0xcc6/0x10c0
Nov 17 09:56:10 LEGION kernel:  ? __check_object_size+0x48/0x3c0
Nov 17 09:56:10 LEGION kernel:  bpf_prog_load+0x803/0x8a0
Nov 17 09:56:10 LEGION kernel:  __sys_bpf+0x404/0x640
Nov 17 09:56:10 LEGION kernel:  x64_sys_call+0x9ab/0x30c0
Nov 17 09:56:10 LEGION kernel:  ? do_syscall_64+0x1d3/0x310
Nov 17 09:56:10 LEGION kernel:  do_syscall_64+0x86/0x310
Nov 17 09:56:10 LEGION kernel:  ? do_syscall_64+0x1d3/0x310
Nov 17 09:56:10 LEGION kernel:  ? refill_obj_stock+0x1a6/0x230
Nov 17 09:56:10 LEGION kernel:  ? update_load_avg+0x1f1/0x840
Nov 17 09:56:10 LEGION kernel:  ? update_curr+0x1c5/0x240
Nov 17 09:56:10 LEGION kernel:  ? generic_exec_single+0x74/0x140
Nov 17 09:56:10 LEGION kernel:  ? kick_ilb+0x17d/0x1d0
Nov 17 09:56:10 LEGION kernel:  ? update_process_times+0x8f/0x140
Nov 17 09:56:10 LEGION kernel:  ? tick_nohz_handler+0xce/0x240
Nov 17 09:56:10 LEGION kernel:  ? __pfx_tick_nohz_handler+0x10/0x10
Nov 17 09:56:10 LEGION kernel:  ? __hrtimer_run_queues+0x22b/0x3e0
Nov 17 09:56:10 LEGION kernel:  ? ktime_get+0x46/0xe0
Nov 17 09:56:10 LEGION kernel:  ? lapic_next_event+0x16/0x20
Nov 17 09:56:10 LEGION kernel:  ? clockevents_program_event+0x9d/0x1e0
Nov 17 09:56:10 LEGION kernel:  ? hrtimer_interrupt+0x138/0x710
Nov 17 09:56:10 LEGION kernel:  ? __sysvec_apic_timer_interrupt+0x4f/0x170
Nov 17 09:56:10 LEGION kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 17 09:56:10 LEGION kernel: RIP: 0033:0x7fa50193864d
Nov 17 09:56:10 LEGION kernel: Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8b e6 0f 00 f7 d8 64 89 01 48
Nov 17 09:56:10 LEGION kernel: RSP: 002b:00007fff9dd68fd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
Nov 17 09:56:10 LEGION kernel: RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fa50193864d
Nov 17 09:56:10 LEGION kernel: RDX: 0000000000000094 RSI: 00007fff9dd690a0 RDI: 0000000000000005
Nov 17 09:56:10 LEGION kernel: RBP: 00007fff9dd69170 R08: 0000000000000000 R09: 0000000000000000
Nov 17 09:56:10 LEGION kernel: R10: 000000000000066c R11: 0000000000000246 R12: 000000009dd69100
Nov 17 09:56:10 LEGION kernel: R13: 00007fff9dd690a0 R14: 000000000000066c R15: 000055ada761b490
Nov 17 09:56:10 LEGION kernel:  </TASK>
Nov 17 09:56:10 LEGION kernel: ---[ end trace 0000000000000000 ]---

As far as I can tell, it still seems to work, but it's a bit worrisome. Could be related to the SCC/control graph changes, and this WARN_ONCE: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/kernel/bpf/verifier.c?h=v6.17.8&id=1cb0f56d96185cb20e63e191fc291191823e6f52.

[hack3ric]

Can confirm on my Arch Linux machine. If it's a verifier bug then it's maybe for the kernel side to fix it? I'm not too familiar with it, but I'll leave this issue open since it is indeed worrisome.

[hack3ric]

Seems like the error message is from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/kernel/bpf/verifier.c?h=v6.17.8&id=9325d53fe9adff354b6a93fda5f38c165947da0f#n2585, unrelated to the WARN_ONCEs. And yet it chose to return success even with this message, which is confusing.

I happen to find out that a recent patch https://lkml.org/lkml/2025/11/7/1313 could be addressing this issue. I don't have time recently to test the kernel, so appreciated if anyone could help :)

[airend]

Seems like the error message is from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/kernel/bpf/verifier.c?h=v6.17.8&id=9325d53fe9adff354b6a93fda5f38c165947da0f#n2585, unrelated to the WARN_ONCEs. And yet it chose to return success even with this message, which is confusing.

Yeah, based on the docs, you'd assume it will outright reject/not load the BPF program altogether, but I cannot see any regressions with handshake, or throughput, which is odd.

I happen to find out that a recent patch https://lkml.org/lkml/2025/11/7/1313 could be addressing this issue. I don't have time recently to test the kernel, so appreciated if anyone could help :)

I will attempt to test this one, but can already confirm that the same bug happens on v6.18-rc6.

[airend]

I happen to find out that a recent patch https://lkml.org/lkml/2025/11/7/1313 could be addressing this issue. I don't have time recently to test the kernel, so appreciated if anyone could help :)

I tested this early patch on top of v6.18-rc6, but no difference, unfortunately.

[hack3ric]

What about this patch set: https://lore.kernel.org/bpf/20251103063108.1111764-1-kafai.wan@linux.dev/ ? It was applied to bpf-next 10+ days ago and not yet applied to mainline.

[airend]

Same verifier bug with https://lore.kernel.org/bpf/20251103063108.1111764-2-kafai.wan@linux.dev/, unfortunately.

[hack3ric]

That's sad. I've now set up CI to test Mimic against various kernel versions, including bpf-next. I'll extract the dmesg it produces after Mimic is loaded to see if they fixed the problem. If they don't, what I could do would be just keep tracking + try to produce a minimal reproducible BPF asm and report it upstream...

[airend]

That's a proper way to debug this. I can tell you that it started with 6.17 since 6.16.y is fine. It could be related to the SCC changes that happened at that point. I didn't try to revert them yet, since those and their follow-up fixes are a bit scattered.

[hack3ric]

Well they didn't fix it. I'll do bisect first then -- when my time is more available.

[airend]

I rebased current bpf-next on top of 6.18.0-rc7 plus added https://lore.kernel.org/lkml/20251028151938.3872003-2-kafai.wan@linux.dev/ since it looked promising. Alas, the verifier bug is still there.

[airend]

Cosmetically, this is the reason for the reg_bounds_sanity_check() verifier bug being displayed now: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0df1a55afa832f463f9ad68ddc5de92230f1bc8a. However, this REG INVARIANTS VIOLATION was likely hit in older kernels too, just hidden away before the verifier_bug() plumbing. I don't know if any bpf/main.c refactoring can avoid this, or it's strictly a spurious verifier bug, but it seems harmless so far. Honestly, mimic has been working really well for me, so I'll just revert this warning for now (bpf-verifier.patch).

[airend]

P.S. This project looks interesting in this context: https://github.com/libbpf/bpfvv (presentation).