Improve thanos querier guide

docs/user-guides/README.md

@@ -2,5 +2,6 @@
 
 * [Using SSA to customize Prometheus](server-side-apply.md)
 * [Federating OpenShift In-Cluster Prometheus](federation.md)
-* [User interface plugins](observability-ui-plugins.md)
+* [User interface (UI) plugins](observability-ui-plugins.md)
 * [Deploying ThanosQuerier for multiple MonitoringStacks](thanos_querier.md)
+* [Perses dashboards](perses-dashboards.md)

docs/user-guides/perses-dashboards.md

@@ -4,9 +4,9 @@ In conjunction with the Monitoring UIPlugin, the Cluster Observability Operator
 
 ## Requirements to display Perses Dashboards in OpenShift Console
 
-- OpenShift 4.15 or later
-- The Cluster Observability Operator 1.2 or later installed and running
-- The Monitoring UIPlugin installed with the perses feature enabled
+- OpenShift 4.15 or later.
+- The Cluster Observability Operator 1.2 or later installed and running.
+- The [Monitoring UIPlugin](https://github.com/rhobs/observability-operator/blob/main/docs/user-guides/observability-ui-plugins.md#plugin-creation-4) installed with the Perses feature enabled.
 
 ## Adding a Perses Dashboard and Datasource
 

docs/user-guides/thanos_querier.md

@@ -14,17 +14,20 @@ pods.
 ## Tutorial
 
 ### Pre-requisites
+
 * Observability operator installed and running.
 * Cluster admin permissions.
 
 ### Installation
 
 We are going to create
-* 1 `MonitoringStack` running in the `project-a` namespace.
-* 1 `MonitoringStack` running in the `project-b` namespace.
-* 1 `ThanosQuerier` running in the `project-c` namespace.
+* 1 `MonitoringStack` running in the `project-a` namespace which monitors API services running in `project-a` and `project-b`.
+* 1 `MonitoringStack` running in the `project-c` namespace which monitors the backend service running in `project-c`.
+* 1 `ThanosQuerier` running in the `project-d` namespace which federates data from the 2 `MonitoringStack`s.
+* Deployments running in `project-a`, `project-b` and `project-c` namespaces which represent a multi-service application.
+* Load generation Deployment running in the `project-d` namespace.
 
-To install the example, run:
+To install the environment, run:
 
 ```shell
 kubectl apply -f docs/user-guides/thanos_querier/install
@@ -33,8 +36,9 @@ kubectl apply -f docs/user-guides/thanos_querier/install
 To verify the installation, run:
 
 ```shell
-kubectl wait --for=condition=Available -A --timeout=10s -l app=example monitoringstacks
+kubectl wait --for=condition=Available -A --timeout=10s -l app.kubernetes.io/part-of monitoringstacks
 kubectl wait --for=condition=Available -A --timeout=10s -l app.kubernetes.io/managed-by=observability-operator deployments
+kubectl wait --for=condition=Available -A --timeout=10s -l app.kubernetes.io/part-of=myapp deployments
 ```
 
 To access the Thanos Query UI, run:
@@ -43,20 +47,20 @@ To access the Thanos Query UI, run:
 kubectl port-forward -n project-c svc/thanos-querier-example 10902:localhost:10902
 ```
 
-Then open `http://localhost:10902` in your browser.
+Then open `http://localhost:10902` in your browser. You can check that all Prometheus instances are present in the Stores page and that metrics are showing up.
 
-### Configuring a dashboard
+### Configuring a Perses dashboard
 
-To install the example dashboard (+datasource), run:
+To install the example Perses dashboard (+datasource), run:
 
 ```shell
-kubectl apply -f docs/user-guides/thanos_querier/dashboard
+kubectl apply -f docs/user-guides/thanos_querier/console
 ```
 
 To verify the installation, run:
 
 ```
-kubectl wait --for=condition=Available uiplugins dashboards
+kubectl wait --for=condition=Available uiplugins monitoring
 ```
 
-You should now be able to access the custom dashboard under `Observe > Dashboards`.
+You should now be able to access the custom dashboard under `Observe > Dashboards (Perses)` in the `project-d` namespace.

docs/user-guides/thanos_querier/console/00-plugin.yaml

@@ -0,0 +1,9 @@
+apiVersion: observability.openshift.io/v1alpha1
+kind: UIPlugin
+metadata:
+  name: monitoring
+spec:
+  type: Monitoring
+  monitoring:
+    perses:
+      enabled: true

docs/user-guides/thanos_querier/console/01-datasource.yaml

@@ -0,0 +1,20 @@
+apiVersion: perses.dev/v1alpha1
+kind: PersesDatasource
+metadata:
+  name: metrics-api
+  namespace: project-d
+  labels:
+    app.kubernetes.io/name: datasource
+    app.kubernetes.io/part-of: monitoring
+spec:
+  config:
+    display:
+      name: "Thanos Querier (myapp)"
+    default: true
+    plugin:
+      kind: PrometheusDatasource
+      spec:
+        proxy:
+          kind: HTTPProxy
+          spec:
+            url: "http://thanos-querier-metrics-api.project-d.svc:10902"

docs/user-guides/thanos_querier/console/03_dashboard.yaml

@@ -0,0 +1,127 @@
+---
+apiVersion: perses.dev/v1alpha1
+kind: PersesDashboard
+metadata:
+  name: app-overview
+  namespace: project-d
+  labels:
+    app.kubernetes.io/name: dashboard
+    app.kubernetes.io/part-of: monitoring
+spec:
+  display:
+    name: Overview
+  variables:
+    - kind: ListVariable
+      spec:
+        display:
+          hidden: false
+        allowAllValue: true
+        allowMultiple: true
+        sort: alphabetical-asc
+        plugin:
+          kind: PrometheusLabelValuesVariable
+          spec:
+            labelName: service
+            matchers:
+              - up{service=~"api|backend"}
+        name: service
+    - kind: ListVariable
+      spec:
+        display:
+          hidden: false
+        allowAllValue: true
+        allowMultiple: true
+        sort: alphabetical-asc
+        plugin:
+          kind: PrometheusLabelValuesVariable
+          spec:
+            labelName: namespace
+            matchers:
+              - up{service=~"$service"}
+        name: namespace
+  panels:
+    "0_0":
+      kind: Panel
+      spec:
+        display:
+          name: Requests rate (req/s)
+        plugin:
+          kind: TimeSeriesChart
+          spec: {}
+        queries:
+          - kind: TimeSeriesQuery
+            spec:
+              plugin:
+                kind: PrometheusTimeSeriesQuery
+                spec:
+                  datasource:
+                    kind: PrometheusDatasource
+                  query: sum by(service, namespace, code) (rate(http_requests_total{service=~"$service",namespace=~"$namespace"}[$__rate_interval]))
+                  seriesNameFormat: svc={{service}},code={{code}},namespace={{namespace}}
+    "0_1":
+      kind: Panel
+      spec:
+        display:
+          name: Errors
+        plugin:
+          kind: TimeSeriesChart
+          spec: {}
+        queries:
+          - kind: TimeSeriesQuery
+            spec:
+              plugin:
+                kind: PrometheusTimeSeriesQuery
+                spec:
+                  datasource:
+                    kind: PrometheusDatasource
+                  query: |-
+                    sum by(service, namespace) (rate(http_requests_total{service=~"$service",namespace=~"$namespace",code!~"2.."}[$__rate_interval]))
+                    /
+                    sum by(service, namespace) (rate(http_requests_total{service=~"$service",namespace=~"$namespace"}[$__rate_interval]))
+                  seriesNameFormat: svc={{service}},namespace={{namespace}}
+    "0_2":
+      kind: Panel
+      spec:
+        display:
+          name: Duration (90th percentile)
+        plugin:
+          kind: TimeSeriesChart
+          spec: {}
+        queries:
+          - kind: TimeSeriesQuery
+            spec:
+              plugin:
+                kind: PrometheusTimeSeriesQuery
+                spec:
+                  datasource:
+                    kind: PrometheusDatasource
+                  query: |-
+                    histogram_quantile(0.9, sum by (le,namespace,service) (rate(http_request_duration_seconds_bucket{service=~"$service",namespace=~"$namespace"}[$__rate_interval])))
+                  seriesNameFormat: svc={{service}},namespace={{namespace}}
+  layouts:
+    - kind: Grid
+      spec:
+        display:
+          title: Rate/Errors/Duration
+          collapse:
+            open: true
+        items:
+          - x: 0
+            "y": 1
+            width: 24
+            height: 7
+            content:
+              $ref: "#/spec/panels/0_0"
+          - x: 0
+            "y": 2
+            width: 24
+            height: 7
+            content:
+              $ref: "#/spec/panels/0_1"
+          - x: 0
+            "y": 3
+            width: 24
+            height: 7
+            content:
+              $ref: "#/spec/panels/0_2"
+  duration: 30m