Enabling SSL caused relay-agent registration to fail.

[britslampe]

This is a cross post from this issue

After applying the -boostrap.yaml for the relay-agent onto clusters that I want to import, the agent is not able to connect to Paralus to register clusters. I did some debugging and found that it was not Okta, but the relay application with this problem.

The certificate generated for SSL was created following the Deploy ClusterIssuer and Certificate Objects with cert-manager.

Expected vs actual behavior

• Expect

  • clusters to register with Paralus and the Cluster Connection status to read SUCCESSFUL when viewing the clusters in a project

• Actual

  • relay-agent unable to connect and register cluster with Paralus due to Method not allowed
  • Error shown:
    [POST /v2/sentry/bootstrap/{templateToken}/register][501] Bootstrap_RegisterBootstrapAgent default &{Code:12 Details:[] Message:Method Not Allowed}

• cluster registration stuck pending and Cluster Connection status reads FAILURE

Steps to reproduce the bug

1. Deploy Paralus
2. Enable SSL
3. Try to import another cluster following the instructions in the Paralus Console

Are you using the latest version of the project?

• chart version: ztka-0.2.4
• app version: v0.2.3

What is your environment setup? Please tell us your cloud provider, operating system, and include the output of kubectl version --output=yaml and helm version. Any other information that you have, eg. logs and custom values, is highly appreciated!

• AWS following the eks-quickstart blog post
• helm v3.12.2
• kubectl v1.25.x

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

• Before adding the certificate I was able to import other clusters and use the kubectl terminal, perhaps the registration function(s) cannot communicate over HTTPS?

• I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
• I'm using the latest version of the project.

[divyanjali1]

@britdm any luck resolving this issue? I ran into same issue while trying to register EKS cluster

[britslampe]

@britdm any luck resolving this issue? I ran into same issue while trying to register EKS cluster

Not yet, but I do wonder if an IAM role is needed here with the correct permissions for the relay agent. EKS clusters can be locked down a bit, even to internal users.

[divyanjali1]

I was able to register the same cluster when SSL is not enabled. Post SSL only I am facing this issue.

I am facing this issue in while importing other kubernetes clusters also, not just EKS

[itcrow]

I had the same problem after enabling SSL. It was overcome when in -boostrap.yaml I manually corrected the connection port from 80 to 443 for

data:
   clusterID: ad8710e7-****-****-****-86b555046661
   relays: '[{"token":"**************","addr":"console.example.com :443","endpoint":"*.core-connector .example.com:443","name":"paralus-core-relay-agent","templateToken":"****************"}]'

console.example.com :80 -> console.example.com :443

[divyanjali1]

@itcrow

My bootstrap YAML has port 443 post enabling SSL , but still facing the error. I am attaching paralus config and relay-agent config.

{"level":"error","ts":"2023-10-17T10:36:55.736Z","caller":"agent/agent.go:397","msg":"Relay Agent::failed to register relay agent error: Post "https://console.paralus.slvr-dig-platsvcmgt.awsdns.internal.das:443/v2/sentry/bootstrap/template%2Fcj3odvnddtem3po9aa7g/register\": tls: failed to verify certificate: x509: certificate signed by unknown authority ","stacktrace":"github.com/paralus/relay/pkg/agent.registerRelayAgent\n\t/build/pkg/agent/agent.go:397\ngithub.com/paralus/relay/pkg/agent.handleRelayNetworks\n\t/build/pkg/agent/agent.go:606"}
{"level":"info","ts":"2023-10-17T10:37:00.728Z","caller":"agent/agent.go:394","msg":"Relay Agent::config: &{TemplateToken:cj3odvnddtem3po9aa7g TemplateName: Scheme:https Mode: Addr:console.paralus.slvr-dig-platsvcmgt.awsdns.internal.das:443 ClientID:ckn668epn18qrmonkqqg ClientIP:100.94.158.146 Name:relay-agent-6556fd4f5f-nzz7j PrivateKey:[] CSR:[] Certificate:[] CACertificate:[] ServerHost: ServerPort:0 Fingerprint:00cf8eaa-1fd8-498d-8c5e-b40a64945ec3} "}
Post "https://console.paralus.slvr-dig-platsvcmgt.awsdns.internal.das:443/v2/sentry/bootstrap/template%2Fcj3odvnddtem3po9aa7g/register": tls: failed to verify certificate: x509: certificate signed by unknown authority
{"level":"error","ts":"2023-10-17T10:37:00.736Z","caller":"agent/agent.go:397","msg":"Relay Agent::failed to register relay agent error: Post "https://console.paralus.slvr-dig-platsvcmgt.awsdns.internal.das:443/v2/sentry/bootstrap/template%2Fcj3odvnddtem3po9aa7g/register\": tls: failed to verify certificate: x509: certificate signed by unknown authority ","stacktrace":"github.com/paralus/relay/pkg/agent.registerRelayAgent\n\t/build/pkg/agent/agent.go:397\ngithub.com/paralus/relay/pkg/agent.handleRelayNetworks\n\t/build/pkg/agent/agent.go:606"}

=================================================================
relay-agent config

paralus-config

[itcrow]

@divyanjali1

tls: failed to verify certificate: x509: certificate signed by unknown authority

You using self-signed certificates (AWS Private CA) ?
I think you need to add the CA certificate to pods' trust root for relay. (https://stackoverflow.com/questions/38968414/kubernetes-add-ca-certificate-to-pods-trust-root)
Or find the appropriate parameter in the relay settings.