BPF token support and testing infrastructure

[dylandreimerink]

This PR introduces test utilities that allows us to write tests for BPF tokens. This RunWithToken utility function is largely based on the test code from the kernel selftests but modified to work with Go and its runtime.

The individual commit messages contain more details. The high level is that we can write tests (and subtests) where we start by calling RunWithToken with a set of delegated permissions. The rest of the test body will be executed in an environment with a BPFFS mounted with those permissions.

This PR also includes BPF token support. This takes the "ambient token" approach, where we do not add any new API surface, but we automatically use tokens when they are available. This allows us to experiment with tokens without committing to an API. We can always add more controls in the future.

Instead of relying on hardcoding search paths or environment variables, this implementation scans the /proc/self/mountinfo procfile to find all BPFFS instances mounted in the processes mount namespace. This means we are able to use any BPFFS path, and even multiple BPFFS'es, for example if one FS has no permissions but holds pinned objects, and a secondary BPFFS is used to the delegated tokens. However, there is no method to select a preferred one, if multiple BPFFS'es with delegated privileges exist (which should be uncommon).

A nice side effect of reading the mount info is that we also know which permissions are delegated which is used in this implementation to enhance error messages when using tokens but the token lacks certain permissions.

The main purpose of this PR was to implement the testing utilities, and the PR is structured such that we can drop the actual token implementation if any of the other proposals is more appealing then this one.

Note: This PR currently has cherry-picked a commit from #1950 for its capability syscalls. This PR should be rebased once #1950 is merged.

[javiercardona-work]

Thanks for doing this. If you end up landing this, I will withdraw #1948.

I tested this PR in our environment, and it works as documented. In case it can be useful, I'm attaching the test program we used: ambient_token_test.go.zip

This runs in our namespaced container with CAP_BPF permissions and a delegated BPF mountpoint (/run/tw/bpf). The mountpoint is automatically detected:

$ /tmp/ambient-token-test            
ambient-token-test v7

=== Diagnostics ===
  user namespace: user:[4026535428]
  unprivileged_bpf_disabled: 1

  Capabilities before:
  CapInh:       0000000000000000
  CapPrm:       000001c7a9ac7dff
  CapEff:       000001c7a9ac7dff
  CapBnd:       000001c7a9ac7dff
  CapAmb:       0000000000000000

=== Token Create ===
  token.Create(): OK (fd=6, mount=/run/tw/bpf)

=== Feature Probes (ambient token) ===
  prog SocketFilter         supported
  prog Kprobe               supported
  prog SchedCLS             supported
  prog XDP                  supported
  prog TracePoint           supported
  prog CGroupSKB            supported
  map  Hash                 supported
  map  Array                supported
  map  PerfEventArray       supported
  map  LRUHash              supported
  map  RingBuf              supported

=== Map Create (ambient token) ===
  map create: OK (fd=3)

=== Program Load (ambient token) ===
  program load: OK (fd=3)

Done.

When running it without CAP_BPF, token creation and all subsequent bpf operations fail, as expected:

$ /tmp/ambient-token-test --drop-caps
ambient-token-test v7

=== Diagnostics ===
  user namespace: user:[4026535428]
  unprivileged_bpf_disabled: 1

  Capabilities before:
  CapInh:       0000000000000000
  CapPrm:       000001c7a9ac7dff
  CapEff:       000001c7a9ac7dff
  CapBnd:       000001c7a9ac7dff
  CapAmb:       0000000000000000

  Dropping CAP_BPF, CAP_SYS_ADMIN, CAP_PERFMON...
  Capabilities after:
  CapInh:       0000000000000000
  CapPrm:       00000107a98c7dff
  CapEff:       00000107a98c7dff
  CapBnd:       00000107a98c7dff
  CapAmb:       0000000000000000

=== Token Create ===
  token.Create(): BPF Token is not available

=== Feature Probes (ambient token) ===
  prog SocketFilter         detect support for SocketFilter: load program: operation not permitted
  prog Kprobe               detect support for Kprobe: load program: operation not permitted
  prog SchedCLS             detect support for SchedCLS: load program: operation not permitted
  prog XDP                  detect support for XDP: load program: operation not permitted
  prog TracePoint           detect support for TracePoint: load program: operation not permitted
  prog CGroupSKB            detect support for CGroupSKB: load program: operation not permitted
  map  Hash                 detect support for Hash: operation not permitted
  map  Array                detect support for Array: operation not permitted
  map  PerfEventArray       detect support for PerfEventArray: operation not permitted
  map  LRUHash              detect support for LRUHash: operation not permitted
  map  RingBuf              detect support for RingBuf: operation not permitted

=== Map Create (ambient token) ===
  map create: FAILED (creating map: map create: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock))

=== Program Load (ambient token) ===
  program load: FAILED (load program: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock))

Done.

[anryko]

Thank you for working on this!

[dylandreimerink]

Rebased on main and updated to match changes made in #1950. Marking ready for review.

[javiercardona-work]

The updated version of this branch still passes our tests in our containerized environment.

[09:47:25 root@tsp_prn/mks/mks-system-pool-jcardona-sparklemuffin-kubelet/0 ~]$ /tmp/ambient-token-test 
ambient-token-test v9

=== Diagnostics ===
  user namespace: user:[4026535252]
  unprivileged_bpf_disabled: 1

  Capabilities before:
  CapInh:       0000000000000000
  CapPrm:       000001c7a9ac7dff
  CapEff:       000001c7a9ac7dff
  CapBnd:       000001c7a9ac7dff
  CapAmb:       0000000000000000

=== Token Create ===
  token.Create(): OK (fd=6, mount=/run/tw/bpf)

=== Feature Probes (ambient token) ===
  prog SocketFilter         supported
  prog Kprobe               supported
  prog SchedCLS             supported
  prog XDP                  supported
  prog TracePoint           supported
  prog CGroupSKB            supported
  map  Hash                 supported
  map  Array                supported
  map  PerfEventArray       supported
  map  LRUHash              supported
  map  RingBuf              supported

=== Map Create (ambient token) ===
  map create: OK (fd=3)

=== Program Load (ambient token) ===
  program load: OK (fd=3)

Done.

[javiercardona-work]

@rgo3 I there something I can do to help merge this PR? Happy to run additional tests on my environment.

[dylandreimerink]

I there something I can do to help merge this PR?

We are waiting for @ti-mo to review, since I made the PR, and he is the only other project maintainer. But Timo is a busy man, with a very long todo list, so this just takes some time.

All reviews are welcome, but Timo's approval determines merge-ability of the PR.

[javiercardona-work]

Hi @ti-mo,

Could you provide a rough estimate on when this might be ready for merge? I’m curious if there are any outstanding design concerns we should address, or if it’s primarily a matter of finding an opening in your schedule. Our team is excited to move away from our internal fork and adopt @dylandreimerink better implementation for this feature, so we’re just looking to coordinate our timeline.

Thanks!
Javier

[ti-mo]

Hi Javier,

This PR is the result of internal discussions, so don't expect any departures from the overall architecture, if that makes things a bit more predictable for you.

For the initial implementation, we want there to not be any API changes, which should make this branch easy to carry in your Cilium fork since there should be no token-related Cilium changes required. I'll start working on this on Monday and it will be merged when it's done.

[ti-mo]

Thanks! This is all I got for now, will do a 2nd round when these are discussed/addressed.

[dylandreimerink]

I figured out that you can request object info on tokens as well, which returns the delegated permissions as numbers, which means we don't have to have all of the from string and parsing logic. I still need to address a few of the outstanding comments with regards to changing the API of the tokens testutils. Just pushing what I have so far.

[ti-mo]

Did another round focusing on the non-testing changes. New RunWithToken looks nice at a glance. 👍

[ti-mo]

@dylandreimerink Played with this a bit after getting back from break.

tl;dr: ErrTokenNotAvailable was removed since it didn't have much purpose if returned *Tokens can be nil. I've completely pushed token handling into func BPF() and added ErrTokenCapabilities to communicate to the caller that tokens are in use and EPERM was encountered. This sentinel is always returned with surrounding context about missing caps. (caveat: wherever possible) This removed a lot of the boilerplate for token plumbing in callers of BPF() and centralized all handling so we don't accidentally miss any call sites.

In response to #1953 (comment):

I ended up moving Token to package sys for this reason. I've also discovered that obj_info for tokens was only added in 6.17, included in the 6.18 LTS release, so the 6.12 LTS won't have the accurate errors we envisioned. I've changed it so that if tok != nil && EPERM && tok.info == nil, we always return ErrTokenCapabilities, even if the token may actually have the right caps, but the kernel throws EPERM somewhere else. I think that's a worthwhile trade-off over re-evaluating the meaning of EPERM across the library; the latter wouldn't be feasible IMO.

I've left the testing harness untouched, but I think we should make some improvements to the subtest output streaming because, at least locally, gotestsum breaks if any of the subtests return errors. We may have to do some stdio parsing to properly pipe the subtest's output through to the parent and to set SKIP/FAIL properly if detected within the subtest. Not sure if we already do something along those lines.

Let's run through the changes offline tomorrow.