Laundry list about benchmarking

[huitseeker]

TL;DR

It would be awesome for bencher if we had two jobs gpu-bench-comparative and bench-comparative with the following properties:

• activate asm on x86_64,
• use or do not use cuda depending on the nature of the called job,
• they use BENCH_OUTPUT=commit-comment by default, because
• they both use message-format=json and pipe to a file named after bench & commit (as gpu-bench-ci does)

In detail

I've been running 6-7 benches (comparative, so 2 runs per) every day, here's the pet peeves outcome:

• if I want to pipe the output (json OR criterion baseline¹) into a tool (resp. critcmp or criterion-table), I have to remember to pass an env variable XXX_BENCH_OUTPUT = "commit-comment",
• one thing we've solved : the name of the var is per-repo,
• the other thing: I have to edit the justfile super often : mostly I run 3 benches in parallel, but I don't have 3 gpu-equipped machines
• there are niceties in the gpu-bench-ci job that are not in the bench job,

Arecibo example:

# Run CPU benchmarks
bench +benches:
  #!/bin/sh
  for bench in {{benches}}; do
    cargo criterion --bench $bench
  done

# Run CUDA benchmarks on GPU
gpu-bench +benches:
  #!/bin/sh
  # The `compute`/`sm` number corresponds to the Nvidia GPU architecture
  # In this case, the self-hosted machine uses the Ampere architecture, but we want this to be configurable
  # See https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/
  export CUDA_ARCH=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | sed 's/\.//g')
  export EC_GPU_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_$CUDA_ARCH --generate-code=arch=compute_$CUDA_ARCH,code=sm_$CUDA_ARCH"
  export EC_GPU_FRAMEWORK="cuda"

  for bench in {{benches}}; do
    cargo criterion --bench $bench --features "cuda"
  done

# Run CUDA benchmarks on GPU, tuned for CI on Linux x86_64
gpu-bench-ci +benches:
  #!/bin/sh
  printenv PATH
  if [ $(uname -m) = "x86_64" ]; then
    FEATURES="cuda,asm"
  else
    FEATURES="cuda"
  fi

  for bench in {{benches}}; do
    cargo criterion --bench $bench --features $FEATURES --message-format=json > "$bench-{{commit}}".json
  done%

• conclusion : I always want the ASM feature, and I most of the time also want json output.
• one thing in the wishlist : if I run one of the jobs that have message-format=json I nearly certainly also want XX_BENCH_OUTPUT = commit-comment

Footnotes

1. cargo bench --bench foo -- --save-baseline foo_at_main ↩