From 4fae606983772e25a709831996b8a21763fd6f06 Mon Sep 17 00:00:00 2001 From: Alexander Greene Date: Fri, 18 Jun 2021 10:09:12 -0400 Subject: [PATCH] Introduce PPROF Documentation This commit introduces documentation detailing how to retrieve PPROF data from the OLM and Catalog operators. Signed-off-by: Per Goncalves da Silva --- .../advanced-tasks/performance-profiling.md | 96 +++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 content/en/docs/advanced-tasks/performance-profiling.md diff --git a/content/en/docs/advanced-tasks/performance-profiling.md b/content/en/docs/advanced-tasks/performance-profiling.md new file mode 100644 index 00000000..f9fc9833 --- /dev/null +++ b/content/en/docs/advanced-tasks/performance-profiling.md @@ -0,0 +1,96 @@ +--- +title: "OLM Performance Profiling Instrumentation" +weight: 10 +description: > + The goal of this document is to familiarize you with the steps to enable and review OLM's performance profiling instrumentation. +--- + +## Prerequisites + +- [go](https://golang.org/dl/) + +## Background + +OLM utilizes the [pprof package](https://golang.org/pkg/net/http/pprof/) from the standard go library to expose performance profiles for the OLM and Catalog Operator. + +Due to the sensitive nature of profiling data, the profiling endpoints will reject any clients that do not present a verifiable certificate. Both operators must be configured with a serving certificate and a client CA bundle in order to access the profiling endpoints. + +This document will dive into the steps to [enable olm performance profiling](#enabling-performance-profiling) and retrieving pprof data from each component. + +## Enabling Performance Profiling + +### Creating a Certificate + +A valid server certificate must be created for each component before the Performance Profiling functionality can be enabled. If you are unfamiliar with certificate generation, we recommend using the [OpenSSL](https://www.openssl.org/) tool-kit and refer to the [request certificate](https://www.openssl.org/docs/man1.1.1/man1/openssl-req.html) documentation. + +Once you have generated a private and public key, this data should be stored in a `TLS Secret`: + +```bash +$ export PRIVATE_KEY_FILENAME=private.key # Replace with the name of the file that contains the private key you generated. +$ export PUBLIC_KEY_FILENAME=certificate.crt # Replace with the name of the file that contains the public key you generated. + +$ kubectl -n my-namespace create secret tls my-name --cert=$PUBLIC_KEY_FILENAME --key=$PRIVATE_KEY_FILENAME +``` + +### Updating OLM to Use the TLS Secret + +Patch the OLM or Catalog Deployment's pod template to use the generated TLS secret: + +- Defining a volume and volumeMount +- Adding the `client-ca`, `tls-key` and `tls-cert` arguments +- Replacing all mentions of port `8080` with `8443` +- Updating the `livenessProbe` and `readinessProbe` to use HTTPS as the scheme + +The steps to patch an existing OLM deployment can be seen below: + +```bash +$ export TLS_SECRET=my-tls-secret +$ export CERT_PATH=/var/run/secrets # Define where to mount the certs. +# Set Deployment name to olm-operator or catalog-operator +$ export DEPLOYMENT_NAME=olm-operator + +$ kubectl patch deployment $DEPLOYMENT_NAME -n olm --type json -p='[ + # Mount the secret to the pod + {"op": "add", "path": "/spec/template/spec/volumes", "value":[{"name": '$TLS_SECRET', "secret": {"secretName": '$TLS_SECRET'}}]}, + {"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts", "value":[{"name": '$TLS_SECRET', "mountPath": '$CERT_PATH'}]}, + + # Add startup arguments + {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--client-ca"}, + {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"}, + {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-key"}, + {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.key"}, + {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-cert"}, + {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"}, + + # Replace port 8080 with 8443 + {"op": "replace", "path": "/spec/template/spec/containers/0/ports/0", "value":{"containerPort": 8443}}, + {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/port", "value":8443}, + {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port", "value":8443}, + + # Update livenessProbe and readinessProbe to use HTTPS + {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/scheme", "value":"HTTPS"}, + {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/scheme", "value":"HTTPS"}, +]' +deployment.apps/olm-operator patched + +``` + +## Accessing PPROF Data + +You will need to be able to access OLM port, for dev purposes the following commands may prove useful: + +```bash +# Set Deployment name to olm-operator or catalog-operator +$ export DEPLOYMENT_NAME=olm-operator +$ kubectl port-forward deployment/$DEPLOYMENT_NAME 8443:8443 -n olm +``` + +You can then curl the OLM `/debug/pprof` endpoint to retrieve default pprof profiles like so: + +```bash +$ curl https://localhost:8443/debug/pprof/heap --cert certificate.crt --key private.key --insecure -o olm-heap + +$ go tool pprof --top olm-heap +``` + +Please review [the official pprof documentation](https://blog.golang.org/pprof) to learn more about pprof.