Webhook fail on internal_api/webhook

[pini-gh]

Hi,

I'm aware that you recommend tilt, but my server runs several services already using ansible + docker-compose, and I'd like it to host a tmate service as well.

I'm almost there: tmate-ssh-server works. But not tmate-websocket where the docker log reports this error on new sessions:

token=[CGM5...] id=9b211210-79b0-11eb-98df-0242ac120007 [info] Session started stoken=CGM5... stoken_ro=ro-bKmp... ssh_only=false foreground=false named=false
[warn] Webhook fail on internal_api/webhook - Retrying event :session_register (:nxdomain)

And the web session URL returns 404 with this text: :(.

I see no error message in the tmate-ssh-server docker log.

Here is my docker-compose configuration:

version: "3.5"

services:
  tmate-ssh-server:
    image: tmate/tmate-ssh-server:latest
    container_name: tmate-ssh-server
    restart: unless-stopped
    cap_add:
    - SYS_ADMIN
    ports:
    - "2200:22"
    environment:
      HAS_WEBSOCKET: 1
      WEBSOCKET_HOSTNAME: tmate-websocket
      SSH_KEYS_PATH: "/etc/tmate-keys"
      SSH_HOSTNAME: tmate.example.com
      SSH_PORT_LISTEN: 22
      SSH_PORT_ADVERTISE: 2200
    volumes:
    - ssh-keys:/etc/tmate-keys
    - unix-sockets:/tmp/tmate/sessions
    networks:
      bridge:

  tmate-websocket:
    image: tmate/tmate-websocket:latest
    container_name: tmate-websocket
    restart: unless-stopped
    environment:
      ERL_NODE_NAME: tmate.example.com
      ERL_COOKIE: myerlcookie
      # Not sure at all about this one
      WEBSOCKET_BASE_URL: http://tmate-websocket:4001/
      USER_FACING_BASE_URL: https://tmate.example.com/
      DAEMON_HMAC_KEY: myhmackey
      INTERNAL_API_AUTH_TOKEN: myinternaltoken
    volumes:
    - unix-sockets:/tmp/tmate/sessions
    networks:
      bridge:

volumes:
  ssh-keys:
  unix-sockets:

networks:
  bridge:

Note: I had to tweak the tmate-ssh-server's entrypoint to pass the websocket container's host name (WEBSOCKET_HOSTNAME) to the daemon, because it was hardcoded to localhost.

[pini-gh]

I eventually managed to have the suite up and running with successful web session. Here is the fixed docker-compose file:

version: "3.5"

services:
  tmate-ssh-server:
    image: tmate/tmate-ssh-server:latest
    container_name: tmate-ssh-server
    restart: unless-stopped
    cap_add:
    - SYS_ADMIN
    ports:
    - "2200:22"
    environment:
      HAS_WEBSOCKET: 1
      WEBSOCKET_HOSTNAME: tmate-websocket
      SSH_KEYS_PATH: "/etc/tmate-keys"
      SSH_HOSTNAME: tmate.example.com
      SSH_PORT_LISTEN: 22
      SSH_PORT_ADVERTISE: 2200
    volumes:
    - ssh-keys:/etc/tmate-keys
    - unix-sockets:/tmp/tmate/sessions
    networks:
      bridge:

  tmate-websocket:
    image: tmate/tmate-websocket:latest
    container_name: tmate-websocket
    restart: unless-stopped
    environment:
      VIRTUAL_HOST: tmate-ws.example.com
      VIRTUAL_PORT: 4001
      CERT_NAME: example.com
      WEBSOCKET_BASE_URL: wss://tmate-ws.example.com/
      MASTER_BASE_URL: https://tmate.example.com/
      USER_FACING_BASE_URL: https://tmate.example.com/
      ERL_NODE_NAME: tmate.example.com
      ERL_COOKIE: myerlcookie
      DAEMON_HMAC_KEY: mydaemonhmackey
      INTERNAL_API_AUTH_TOKEN: myinternalapiauthtoken
    volumes:
    - unix-sockets:/tmp/tmate/sessions
    networks:
      bridge:

  tmate-master:
    image: tmate/tmate-master:latest
    container_name: tmate-master
    # https://github.com/tmate-io/tmate-master/issues/13#issuecomment-787470427
    hostname: tmate-1
    domainname: example.com
    restart: unless-stopped
    expose:
    - 4000
    environment:
      VIRTUAL_HOST: tmate.example.com
      VIRTUAL_PORT: 4000
      CERT_NAME: example.com
      MASTER_BASE_URL: https://tmate.example.com/
      ERL_COOKIE: myerlcookie
      # Must match <hostname>.<domainname>
      ERL_NODE_NAME: tmate-1.example.com
      # ** (ArgumentError) cookie store expects conn.secret_key_base to be at least 64 bytes
      SECRET_KEY_BASE: 0123456789abcdefghijklmnopqrstuvwxyz=0123456789abcdefghijklmnopqrstuvwxyz
      INTERNAL_API_AUTH_TOKEN: myinternalapiauthtoken
      PG_URI: postgres://tmate:postgrespassword@tmate-db:5432/tmate
      # Muste be >= 2
      PG_POOLSIZE: 5
    networks:
      bridge:
      internal:

  tmate-db:
    image: postgres:12-alpine
    container_name: tmate-db
    restart: unless-stopped
    environment:
      POSTGRES_USER: tmate
      POSTGRES_PASSWORD: postgrespassword
      POSTGRES_DB: tmate
    volumes:
    - pgdata:/var/lib/postgresql/data
    networks:
      internal:

volumes:
  ssh-keys:
  unix-sockets:
  pgdata:

networks:
  bridge:
  internal:
    internal: true

Note: the variables VIRTUAL_HOST, VIRTUAL_PORT and CERT_NAME are related to the reverse proxy nginx-proxy.

[giuseppegambino92]

I eventually managed to have the suite up and running with successful web session. Here is the fixed docker-compose file:

version: "3.5"

services:
  tmate-ssh-server:
    image: tmate/tmate-ssh-server:latest
    container_name: tmate-ssh-server
    restart: unless-stopped
    cap_add:
    - SYS_ADMIN
    ports:
    - "2200:22"
    environment:
      HAS_WEBSOCKET: 1
      WEBSOCKET_HOSTNAME: tmate-websocket
      SSH_KEYS_PATH: "/etc/tmate-keys"
      SSH_HOSTNAME: tmate.example.com
      SSH_PORT_LISTEN: 22
      SSH_PORT_ADVERTISE: 2200
    volumes:
    - ssh-keys:/etc/tmate-keys
    - unix-sockets:/tmp/tmate/sessions
    networks:
      bridge:

  tmate-websocket:
    image: tmate/tmate-websocket:latest
    container_name: tmate-websocket
    restart: unless-stopped
    environment:
      VIRTUAL_HOST: tmate-ws.example.com
      VIRTUAL_PORT: 4001
      CERT_NAME: example.com
      WEBSOCKET_BASE_URL: wss://tmate-ws.example.com/
      MASTER_BASE_URL: https://tmate.example.com/
      USER_FACING_BASE_URL: https://tmate.example.com/
      ERL_NODE_NAME: tmate.example.com
      ERL_COOKIE: myerlcookie
      DAEMON_HMAC_KEY: mydaemonhmackey
      INTERNAL_API_AUTH_TOKEN: myinternalapiauthtoken
    volumes:
    - unix-sockets:/tmp/tmate/sessions
    networks:
      bridge:

  tmate-master:
    image: tmate/tmate-master:latest
    container_name: tmate-master
    # https://github.com/tmate-io/tmate-master/issues/13#issuecomment-787470427
    hostname: tmate-1
    domainname: example.com
    restart: unless-stopped
    expose:
    - 4000
    environment:
      VIRTUAL_HOST: tmate.example.com
      VIRTUAL_PORT: 4000
      CERT_NAME: example.com
      MASTER_BASE_URL: https://tmate.example.com/
      ERL_COOKIE: myerlcookie
      # Must match <hostname>.<domainname>
      ERL_NODE_NAME: tmate-1.example.com
      # ** (ArgumentError) cookie store expects conn.secret_key_base to be at least 64 bytes
      SECRET_KEY_BASE: 0123456789abcdefghijklmnopqrstuvwxyz=0123456789abcdefghijklmnopqrstuvwxyz
      INTERNAL_API_AUTH_TOKEN: myinternalapiauthtoken
      PG_URI: postgres://tmate:postgrespassword@tmate-db:5432/tmate
      # Muste be >= 2
      PG_POOLSIZE: 5
    networks:
      bridge:
      internal:

  tmate-db:
    image: postgres:12-alpine
    container_name: tmate-db
    restart: unless-stopped
    environment:
      POSTGRES_USER: tmate
      POSTGRES_PASSWORD: postgrespassword
      POSTGRES_DB: tmate
    volumes:
    - pgdata:/var/lib/postgresql/data
    networks:
      internal:

volumes:
  ssh-keys:
  unix-sockets:
  pgdata:

networks:
  bridge:
  internal:
    internal: true

Note: the variables VIRTUAL_HOST, VIRTUAL_PORT and CERT_NAME are related to the reverse proxy nginx-proxy.

I have this error:

tmate-ssh-server | sh: out of range
tmate-ssh-server | fatal: Error listening to socket: ECDSA, ED25519, DSA, or RSA host key file must be set\n

[pini-gh]

I have this error:

tmate-ssh-server | sh: out of range
tmate-ssh-server | fatal: Error listening to socket: ECDSA, ED25519, DSA, or RSA host key file must be set\n

I guess you didn't configure the host keys in the ssh-keys volume.

[giuseppegambino92]

I have this error:
tmate-ssh-server | sh: out of range
tmate-ssh-server | fatal: Error listening to socket: ECDSA, ED25519, DSA, or RSA host key file must be set\n

I guess you didn't configure the host keys in the ssh-keys volume.

Resolve. Thanks!!
I have another problem, going on my domain, on port 4000, I have no service listening. I have also activated nginx-proxy.
Another error I detect is:
tmate-ssh-server | fatal: Cannot connect to websocket server at localhost:4002

This is my file:

docker-compose.txt

[pini-gh]

I have another problem, going on my domain, on port 4000, I have no service listening. I have also activated nginx-proxy.

Then you should use the port on whitch nginx-proxy is listening : 80 or 443. The requests should be passed to the proper port of the proper docker instance depending on the requested domaine name.

tmate-ssh-server | fatal: Cannot connect to websocket server at localhost:4002

Might be because you define both HAS_WEBSOCKET and WEBSOCKET_HOSTNAME. These variables are mutually exclusive, the former beeing the same as WEBSOCKET_HOSTNAME=localhost.

[giuseppegambino92]

I have another problem, going on my domain, on port 4000, I have no service listening. I have also activated nginx-proxy.

Then you should use the port on whitch nginx-proxy is listening : 80 or 443. The requests should be passed to the proper port of the proper docker instance depending on the requested domaine name.

tmate-ssh-server | fatal: Cannot connect to websocket server at localhost:4002

Might be because you define both HAS_WEBSOCKET and WEBSOCKET_HOSTNAME. These variables are mutually exclusive, the former beeing the same as WEBSOCKET_HOSTNAME=localhost.

Should I use something like this?
docker run -d -p 4000:4000 -e DEFAULT_HOST = foo.bar.com -v /var/run/docker.sock:/tmp/docker.sock:ro nginxproxy / nginx-proxy

Sorry, but I'm having a hard time configuring the variables
VIRTUALHOST (I believe it is defined at the launch of nginx)
VIRTUAL_PORT
CERT_NAME
ERL_NODE_NAME
MASTER_BASE_URL
USER_FACING_BASE_URL

Could you explain the meaning to me more clearly?

Very last question:
In the tmate-master container there is the hostname and the domainname, in this case I have to put as domain name (weatheritaly.it) but as hostname, what should I enter?

Thanks in advance for your patience.

[giuseppegambino92]

I have another problem, going on my domain, on port 4000, I have no service listening. I have also activated nginx-proxy.

Then you should use the port on whitch nginx-proxy is listening : 80 or 443. The requests should be passed to the proper port of the proper docker instance depending on the requested domaine name.

tmate-ssh-server | fatal: Cannot connect to websocket server at localhost:4002

Might be because you define both HAS_WEBSOCKET and WEBSOCKET_HOSTNAME. These variables are mutually exclusive, the former beeing the same as WEBSOCKET_HOSTNAME=localhost.

Should I use something like this?
docker run -d -p 80:80 -e DEFAULT_HOST = foo.bar.com -v /var/run/docker.sock:/tmp/docker.sock:ro nginxproxy / nginx-proxy

Sorry, but I'm having a hard time configuring the variables
VIRTUALHOST (I believe it is defined at the launch of nginx)
VIRTUAL_PORT
ERL_NODE_NAME
MASTER_BASE_URL
USER_FACING_BASE_URL

Could you explain the meaning to me more clearly?

Very last question:
In the tmate-master container there is the hostname and the domainname, in this case I have to put as domain name (weatheritaly.it) but as hostname, what should I enter?

Thanks in advance for your patience.

I have this error.

tmate-master | 20:23:40.016 [info] [swarm on tmate@tmate-1.example.com] [tracker:cluster_wait] no connected nodes, proceeding without sync
nginx-proxy_1 | nginx.1 | 2021/09/10 20:23:40 [error] 45#45: *1 no live upstreams while connecting to upstream, client: 172.26.0.1, server: tmate-ws.example.com, request: "GET / HTTP/1.1", upstream: "http://tmate-ws.example.com/", host: "tmate-ws.example.com"
nginx-proxy_1 | nginx.1 | tmate-ws.example.com 172.26.0.1 - - [10/Sep/2021:20:23:40 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.74.0" "tmate-ws.example.com"
nginx-proxy_1 | nginx.1 | 2021/09/10 20:23:42 [error] 45#45: *2 no live upstreams while connecting to upstream, client: 172.26.0.1, server: tmate-ws.example.com, request: "GET / HTTP/1.1", upstream: "http://tmate-ws.example.com/", host: "tmate-ws.example.com"
nginx-proxy_1 | nginx.1 | tmate-ws.example.com 172.26.0.1 - - [10/Sep/2021:20:23:42 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.74.0" "tmate-ws.example.com"
nginx-proxy_1 | nginx.1 | 2021/09/10 20:23:45 [error] 45#45: *3 no live upstreams while connecting to upstream, client: 172.26.0.1, server: tmate.example.com, request: "GET / HTTP/1.1", upstream: "http://tmate.example.com/", host: "tmate.example.com"
nginx-proxy_1 | nginx.1 | tmate.example.com 172.26.0.1 - - [10/Sep/2021:20:23:45 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.74.0" "tmate.example.com"
docker-compose.txt

Attach my configuration
docker-compose.txt

[pini-gh]

Should I use something like this?
docker run -d -p 4000:4000 -e DEFAULT_HOST = foo.bar.com -v /var/run/docker.sock:/tmp/docker.sock:ro nginxproxy / nginx-proxy

No. Please read carefuly the nginx-proxy documentation and try first with simple examples such as dockerbogo/docker-nginx-hello-world. I recommend you use the acme-companion as well to ease HTTPS certificates handling. Once you're at ease with these components, you should be able to fix your tmate configuration.

[giuseppegambino92]

Should I use something like this?
docker run -d -p 4000:4000 -e DEFAULT_HOST = foo.bar.com -v /var/run/docker.sock:/tmp/docker.sock:ro nginxproxy / nginx-proxy

No. Please read carefuly the nginx-proxy documentation and try first with simple examples such as dockerbogo/docker-nginx-hello-world. I recommend you use the acme-companion as well to ease HTTPS certificates handling. Once you're at ease with these components, you should be able to fix your tmate configuration.

Thanks for all.

Now i have this error. What is?

mate-ssh-server | [JgFW...] sysname=Linux machine=armv7l release=5.10.60-v7l+ version=#1449 SMP Wed Aug 25 15:00:44 BST 2021 nodename=raspberrypi
tmate-master | 15:06:35.994 [info] GET /internal_api/named_session_prefix
nginx-proxy_1 | nginx.1 | www.weatheritaly.it 172.26.0.1 - - [11/Sep/2021:15:06:35 +0000] "GET /internal_api/named_session_prefix?api_key=tmk-ABekHdcCJOl14omLBWcoGWYFlV HTTP/1.1" 200 18 "-" "hackney/1.15.2" "172.26.0.6:4000"
tmate-master | 15:06:35.998 [info] Sent 200 in 3ms
tmate-websocket | [error] GenServer #PID<0.1337.0> terminating
tmate-websocket | ** (FunctionClauseError) no function clause matching in Access.get/3
tmate-websocket | (elixir) lib/access.ex:265: Access.get("{"prefix":"test/"}", "prefix", nil)
tmate-websocket | (tmate) lib/tmate/master_api.ex:32: Tmate.MasterApi.get_named_session_prefix/1
tmate-websocket | (tmate) lib/tmate/session.ex:225: Tmate.Session.get_named_session_tokens/3
tmate-websocket | (tmate) lib/tmate/session.ex:276: Tmate.Session.finalize_session_init/1
tmate-websocket | (tmate) lib/tmate/session.ex:131: Tmate.Session.handle_call/3
tmate-websocket | (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
tmate-websocket | (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
tmate-websocket | (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
tmate-websocket | Last message (from #PID<0.1336.0>): {:notify_daemon_msg, [1, '\t']}
tmate-websocket | State: %{clients: %{}, current_layout: [], daemon: {Tmate.DaemonTcp, {#PID<0.1336.0>, #Port<0.17>, :ranch_tcp}}, daemon_protocol_version: 6, foreground: true, init_state: %{client_version: "2.4.0", ip_address: "79.56.173.108", named_session: %{api_key: "tmk-ABekHdcCJOl14omLBWcoGWYFlV", ro: nil, rw: "test"}, reconnection_data: nil, ssh_cmd_fmt: "ssh -p2200 %s@www.weatheritaly.it", stoken: "JgFWWfjchGK25vdShT8yAz6hX", stoken_ro: "ro-zPXKY9Ys2a2BrMVUvq4mutLAH", uname: %{machine: "armv7l", nodename: "raspberrypi", release: "5.10.60-v7l+", sysname: "Linux", version: "#1449 SMP Wed Aug 25 15:00:44 BST 2021"}, user_webhook_opts: [url: nil, userdata: ""]}, initialized: false, pending_ws_subs: [], registry: {Tmate.SessionRegistry, Tmate.SessionRegistry}, ssh_only: false, webhook_pids: [], webhooks: [{Tmate.Webhook, [url: "http://www.weatheritaly.it/internal_api/webhook", userdata: "myinternalapiauthtoken"]}], ws_subs: []}
tmate-websocket | Client #PID<0.1336.0> is alive
tmate-websocket |
tmate-websocket | (stdlib) gen.erl:167: :gen.do_call/4
tmate-websocket | (elixir) lib/gen_server.ex:1006: GenServer.call/3
tmate-websocket | (tmate) lib/tmate/daemon_tcp.ex:41: Tmate.DaemonTcp.receive_data/2
tmate-websocket | (tmate) lib/tmate/daemon_tcp.ex:25: Tmate.DaemonTcp.handle_info/2
tmate-websocket | (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
tmate-websocket | (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
tmate-websocket | (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
tmate-websocket | [error] Ranch listener :daemon_tcp had connection process started with Tmate.DaemonTcp:start_link/4 at #PID<0.1336.0> exit with reason: {:function_clause, [{Access, :get, ["{"prefix":"test/"}", "prefix", nil], [file: 'lib/access.ex', line: 265]}, {Tmate.MasterApi, :get_named_session_prefix, 1, [file: 'lib/tmate/master_api.ex', line: 32]}, {Tmate.Session, :get_named_session_tokens, 3, [file: 'lib/tmate/session.ex', line: 225]}, {Tmate.Session, :finalize_session_init, 1, [file: 'lib/tmate/session.ex', line: 276]}, {Tmate.Session, :handle_call, 3, [file: 'lib/tmate/session.ex', line: 131]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 661]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 690]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}
tmate-websocket |
tmate-ssh-server | [JgFW...] fatal: Connection to websocket server closed
tmate-master | 15:06:36.286 [info] [swarm on tmate@tmate-1.example.com] [tracker:cluster_wait] joining cluster..
tmate-master | 15:06:36.286 [info] [swarm on tmate@tmate-1.example.com] [tracker:cluster_wait] no connected nodes, proceeding without sync

[pini-gh]

No idea.