Running Grafana in OCP 4.16

[izderadicka]

I'm following this guide content/o11y/ocp-grafana - https://cloud.redhat.com/experts/o11y/ocp-grafana/

It's bit outdated, but updates are more or less straightforward, but I got stuck at the very end.

All requests to created Prometheus datasource ends with 401 Unauthorized .

The service account for grafana is actually grafana-grafana-cr-sa it does have cluster role cluster-monitoring-view bound as described.

What confuses me, is that guide is talking about basic auth, but with updated helm chart OAuth proxy is actually used so login as my Openshift user. But my user is cluster-admin, so I guess it should have access everywhere.

Data source is defined like this:

kind: GrafanaDatasource
metadata:
  name: prometheus-grafanadatasource
spec:
  datasource:
    access: proxy
    editable: true
    isDefault: true
    jsonData:
        httpHeaderName1: 'Authorization'
        timeInterval: 5s
        tlsSkipVerify: true
    name: Prometheus
    secureJsonData:
        httpHeaderValue1: 'Bearer ${BEARER_TOKEN}'
    type: prometheus
    url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
  instanceSelector:
    matchLabels:
      dashboards: grafana

Where BEARER token is generated from service account.

I'd like to connect Grafana to OCP system metrics on thanos querier, but cannot get through. Also tried bit different approach, with direct deployment of grafana, which did worked well for Loki and logs, but again got stuct there on metrics - that time getting 403 responses for the datasource.

Is there somewhere recent example like this one, which will work with OCP 4.16 and recent versions of Grafana.

Thanks