Fix bench-deploy.yml action

[samuelburnham]

Changes

• Fixes deployment of historical benchmarks in caller repos
• Adds support for labeled points on the historical graph

Overview

The following is meant to elucidate the current state of the historical plotting workflow for the benefit of reviewers and myself. It should evolve into inline or hosted documentation at some point.

Historical plot data

The core file structure is a benchmarks/history directory on the gh-pages branch that contains:

• Historical data .tar.gz archives, one for each commit or workflow run, which contain the Criterion benchmark results and Cargo.lock for the given commit.
• Historical data in plot-data.json, which contains only the relevant metadata and average benchmark result for each of the. This file is persistent and append-only, and if it's not found then it is re-created using each of the historical .tar.gz results.
• .png plot images, created on each run using plot-data.json
• HTML to render the images.

This is all created/deployed by the workflow after running the benchmarks, with the only prerequisite being an existing gh-pages branch deployed via GitHub Pages. See https://github.com/lurk-lab/ci-lab/tree/gh-pages and the Successful run below as an example

Benchmark data format

This workflow expects a specific Criterion Benchmark ID format in order to parse and plot data correctly. The schema used by Criterion is as follows:

<bench_group>/<bench_name>/<bench_params>

This ID is then printed to stdout and stored in the resulting benchmark JSON, along with the bench results and other statistics.

For the purpose of gh-pages benchmarks, bench_params must be equivalent to <commit_hash>-<commit_timestamp>-<params>, so it will look like the following example in Lurk:

Fibonacci-num=10/Prove/dd2a8e6-2024-02-20T22:48:21-05:00-rc=100

When plotting, we split the data into the following groups:

• One plot per bench_group/bench_name pair, e.g. Fibonacci-num=10-Prove
• Each (x, y) coordinate is a pair of (commit_timestamp, bench_result), optionally labeled with the commit_hash
• Each line of (x, y) coordinates is grouped by bench_params, e.g. one line each for rc=100, rc=200

A test plot can be viewed at https://lurk-lab.github.io/ci-lab/benchmarks/history/plots.html, though the data is still a WIP.

Next steps

• Test with lurk-rs and arecibo once merged. This will require formatting gh-pages benchmarks properly using the <LURK|ARECIBO>_BENCH_OUTPUT env var and the new schema shown above. Also, this PR breaks plotting compatibility with old benchmarks so they will have to be manually moved to another directory (e.g. benchmarks/history/deprecated)
• JSON conversion script between gh-pages and commit-comment (see bench-deploy: Convert between Criterion JSON formats #52)
• Massage old JSON benchmark data to work with new benchmark data format. As seen in https://lurk-lab.github.io/ci-lab/benchmarks/history/plots.html, our old data doesn't follow the current data format and must be manually updated or deprecated. In my testing, updating old benchmark formats proved quite challenging due to the multiple old formats and attributes to change. More discussion is needed but I'm leaning toward archiving them for now and later extracting a simpler data set (similar to `plot-data.json) for plot integration.
• Switch plots from .png to .svg to enable zooming in/out

Successful run

https://github.com/lurk-lab/ci-lab/actions/runs/8074420081/job/22060261452
https://lurk-lab.github.io/ci-lab/benchmarks/history/plots.html

[huitseeker]

All the changes in there look reasonable so far. Looking forward to the full PR!