x509: certificate signed by unknown authority

[ehogeweg]

Hello,

I have followed the steps in the article to set up a fleet-server, but for some reason the fleet-server does not like the self signed certificate. I double checked all the environment variable flags and they appear correct. Or at least identical to the gitlab repo.

Any suggestions would be greatly appreaciated.

Cheers,

Erwin

FLEET env vars:

# env | grep ^FLEET_
FLEET_SERVER_POLICY_ID=fleet-server-policy
FLEET_INSECURE=true
FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
FLEET_SERVER_INSECURE_HTTP=true
FLEET_ENROLL=1
FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
FLEET_SERVER_ELASTICSEARCH_INSECURE=true
FLEET_URL=https://fleet-server:8220
FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
FLEET_SERVER_ENABLE=1
FLEET_CA=/certs/ca/ca.crt

fleet-server log

{"log.level":"info","@timestamp":"2023-12-04T00:11:11.363Z","message":"request accepted","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":61,"file.name":"middleware/log_middleware.go"},"service.name":"apm-server","user_agent.original":"apm-agent-python/6.17.0 (my_python_service)","http.request.id":"ebe88572-690c-4c26-ae07-8b9e938434d6","log.logger":"request","http.request.method":"POST","url.original":"/intake/v2/events","event.duration":711917,"http.response.status_code":202,"ecs.version":"1.6.0","source.address":"172.26.0.2","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-12-04T00:11:12.683Z","message":"precondition failed: x509: certificate signed by unknown authority","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":64,"file.name":"beater/waitready.go"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-12-04T00:11:14.449Z","message":"Failed to connect to backoff(elasticsearch(https://es01:9200)): Get \"https://es01:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":150,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-12-04T00:11:14.449Z","message":"Attempting to reconnect to backoff(elasticsearch(https://es01:9200)) with 4 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":141,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-12-04T00:11:14.456Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"network":"tcp","address":"es01:9200","ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":38,"file.name":"transport/logging.go"},"service.name":"metricbeat","ecs.version":"1.6.0"}

[ehogeweg]

Disregard this issue. In the end it DID turn out to be a formatting error in the certificate yml.

[olitooni]

Yeah I was fooled by the indentation in the yaml example for Advanced YAML configuration from Getting started with the Elastic Stack and Docker Compose: Part 2.

ssl:
certificate_authorities:
- |

Should be:

ssl:
  certificate_authorities:
    - |

[timhuebener]

thanks @olitooni and @ehogeweg!

For any future people following the guide:
Another error source for me were old certs. So, I also removed all es-cluster prefixed docker volumes like es-cluster_certs.

[blinkomaniak]

Yeah I was fooled by the indentation in the yaml example for Advanced YAML configuration from Getting started with the Elastic Stack and Docker Compose: Part 2.

ssl:
certificate_authorities:
- |

Should be:

ssl:
  certificate_authorities:
     - |

Hi Are you sure there must be 5 spaces for the "- |" ?

[olitooni]

Yeah I was fooled by the indentation in the yaml example for Advanced YAML configuration from Getting started with the Elastic Stack and Docker Compose: Part 2.

ssl:
certificate_authorities:
- |

Should be:

ssl:
  certificate_authorities:
     - |

Hi Are you sure there must be 5 spaces for the "- |" ?

Ah sorry. Corrected with another error. Fixed it now.

[blinkomaniak]

thanks @olitooni and @ehogeweg!

For any future people following the guide: Another error source for me were old certs. So, I also removed all es-cluster prefixed docker volumes like es-cluster_certs.

Do I just need to remove the variable "COMPOSE_PROJECT_NAME"? Can you please elaborate a little more on how the prefix affects the certs? Sorry, but for me is not quite clear what you want to transmit

[timhuebener]

When settings things up, docker creates volumes for the certs. However, when you mess up and have to create new certificates you first have to remove the old certs volumes otherwise you keep using the old ones and no new ones are generated.

[BertV1]

just my two cents, if you start the docker-compose project, and you've made an error in your cert config:

• without having to completely remove all your containers,
• you can prune the affected volumes,
• edit your cert related actions in the docker-compose yml
• restart the cluster

[blinkomaniak]

When settings things up, docker creates volumes for the certs. However, when you mess up and have to create new certificates you first have to remove the old certs volumes otherwise you keep using the old ones and no new ones are generated.

Oh I see. That's why the Fleet Server never "integrates" into Kibana because the ca.crt is not "updated". Thanks!

[robvandyck]

Thank you for this great tutorial!
I did not get this working using the suggested method.

I did get it working by directly pointing to the certificate in the YAML config:
ssl.certificate_authorities: ["/certs/ca/ca.crt"]

[robvandyck]

Since this was asked directly to me: I did this in the user interface.

[jooyyy]

ssl.certificate_authorities: ["/certs/ca/ca.crt"]

Very nice!!!

[jeffatfw]

I am using this setup with STACK_VERSION=8.15.0 and I was getting this error in the logs of the fleet-server:
"message":"no CA certificate matching the fingerprint","component":

To fix this I followed what was discussed here: https://discuss.elastic.co/t/elastic-agent-ca-trusted-fingerprint-does-not-working-but-the-certificate-works/362239/3

The output edit fields that worked to resolve that error:

[Zapfmeister]

The automation way would be to add the following to the kibana.yml:

xpack.fleet.outputs:
  - id: elasticsearch
    name: Elasticsearch
    type:  elasticsearch
    hosts: ["https://es01:9200"]
    is_default: true
    config:
      ssl:
        certificate_authorities: ["/certs/ca/ca.crt"]

I also added a pull request:
#7