Benchmark-Runner

This tool provides a lightweight and flexible framework for running benchmark workloads on Kubernetes/OpenShift Pod or VM.

This tool support the following workloads:

• hammerdb: running hammerdb workload on the following databases: MSSQL, Mariadb, Postgresql on Pod and VM with Configuration
• stressng: running stressng workload on Pod or VM with Configuration
• uperf: running uperf workload on Pod or VM with Configuration

** First Phase: supports benchmark-operator workloads

Benchmark-runner grafana dashboard example:

Reference:

• The benchmark-runner package is located in PyPi
• The benchmark-runner container image is located in Quay.io

Documentation

Documentation is available at benchmark-runner.readthedocs.io

Table of Contents

• Run workload using Podman or Docker
• Run workload in Pod using Kubernetes or OpenShift
• Grafana dashboards
• How to add new workload

Run workload using Podman or Docker

Environment variables description:

mandatory: WORKLOAD=$WORKLOAD

Choose one from the following list:

['stressng_pod', 'stressng_vm', 'stressng_kata','uperf_pod', 'uperf_vm', 'uperf_kata', 'hammerdb_pod_mariadb', 'hammerdb_pod_mssql', 'hammerdb_pod_postgres', 'hammerdb_vm_mariadb', 'hammerdb_vm_mssql', 'hammerdb_vm_postgres', 'hammerdb_kata_mariadb', 'hammerdb_kata_mssql', 'hammerdb_kata_postgres']

auto: NAMESPACE=benchmark-operator [ The default namespace is benchmark-operator ]

auto: OCS_PVC=True [ True=OCS PVC storage, False=Ephemeral storage, default True ]

auto: SYSTEM_METRICS=True [ True=collect metric, False=not collect metrics, default True ]

auto: RUNNER_PATH=/ [ The default work space is / ]

optional: KUBEADMIN_PASSWORD=$KUBEADMIN_PASSWORD

optional: PIN_NODE_BENCHMARK_OPERATOR=$PIN_NODE_BENCHMARK_OPERATOR [node selector for benchmark operator pod]

optional: PIN_NODE1=$PIN_NODE1 [node1 selector for running the workload]

optional: PIN_NODE2=$PIN_NODE2 [node2 selector for running the workload, i.e. uperf server and client, hammerdb database and workload]

optional: ELASTICSEARCH=$ELASTICSEARCH [ elasticsearch service name]

optional: ELASTICSEARCH_PORT=$ELASTICSEARCH_PORT

podman run --rm -e WORKLOAD=$WORKLOAD -e KUBEADMIN_PASSWORD=$KUBEADMIN_PASSWORD -e PIN_NODE_BENCHMARK_OPERATOR=$PIN_NODE_BENCHMARK_OPERATOR -e PIN_NODE1=$PIN_NODE1 -e PIN_NODE2=$PIN_NODE2 -e ELASTICSEARCH=$ELASTICSEARCH -e ELASTICSEARCH_PORT=$ELASTICSEARCH_PORT -e log_level=INFO -v $KUBECONFIG:/root/.kube/config --privileged quay.io/ebattat/benchmark-runner:latest

Run workload in Pod using Kubernetes or OpenShift

[TBD]

Grafana dashboards

There are 3 grafana dashboards templates:

1. benchmark-runner-ci-status-report.json
2. benchmark-runner-report.json
3. benchmark-runner-compare-3-ci-dates-report.json
4. benchmark-runner-compare-3-ocp-versions-report.json
5. system-metrics-report.json

** After importing json in grafana, need to configure elasticsearch data source. (for more details: see HOW_TO.md)

How to add new workload

see HOW_TO.md