Peer certificate verification failed #7362
Description
2025-10-15T17:05:17.298858Z 0 [info ] CCF/src/host/time_bound_logger.h:54 | Operation took too long (523.911ms): Writing ledger entry - 315 bytes, committable=false
2025-10-15T17:05:17.346035Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:17.346111Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:17.351842Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:17.351904Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:17.357952Z 0 [fail ] CCF/src/node/channels.h:630 | <- n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] (INITIATED): Peer certificate verification failed - recv_key_exchange_response failed to verify signature from cert:
-----BEGIN CERTIFICATE-----
MIIByjCCAVCgAwIBAgIQCAq/FFLx5pF8BGJkm9VtSjAKBggqhkjOPQQDAzAbMRkw
FwYDVQQDDBBDQ0YgVGVzdCBTZXJ2aWNlMB4XDTI1MTAxNTE3MDQ1MVoXDTI1MTAx
NjE3MDQ1MFowEzERMA8GA1UEAwwIQ0NGIE5vZGUwdjAQBgcqhkjOPQIBBgUrgQQA
IgNiAASpcqspAmBKeInPnO0ElZTnRH4W+iyWAixULDSlzud7rQ8rYXVr5KjQ5wxY
KmlkkB86sYFlL+VHck7PzNZe5h+5r3JXXz1d8S225rEuPMCzB5YKomP27BPeppMW
pbwJoqujYTBfMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFDq7nzJwAk5iUJEEWHGM
NlH/80GDMB8GA1UdIwQYMBaAFGND18u2OwzKb8uQnEqSNKfNn+j/MA8GA1UdEQQI
MAaHBH+1qbkwCgYIKoZIzj0EAwMDaAAwZQIxAPpJUE/HXao5tn1pXDGbyouMLjkS
pW62kXFZMwUjuYtFfj2g2i4FT7SCYPGk/2WkLgIwASLDPxrOYglV7LHMvXARCLJH
7jYhwbJMqCcpowED5gxnMJ41F9L07eXh6Xx3vdHh
-----END CERTIFICATE-----
2025-10-15T17:05:17.358121Z 0 [info ] CCF/src/node/channels.h:1094 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] cannot receive encrypted message: not established a receive key, status=INITIATED
2025-10-15T17:05:17.358303Z 0 [fail ] CCF/src/node/rpc/forwarder.h:540 | Exception in void ccf::Forwarder<ccf::NodeToNode>::recv_message(const ccf::NodeId &, const uint8_t *, size_t) [ChannelProxy = ccf::NodeToNode]
2025-10-15T17:05:17.358477Z 0 [info ] CCF/src/node/channels.h:1094 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] cannot receive encrypted message: not established a receive key, status=INITIATED
2025-10-15T17:05:17.358528Z 0 [fail ] CCF/src/node/rpc/forwarder.h:540 | Exception in void ccf::Forwarder<ccf::NodeToNode>::recv_message(const ccf::NodeId &, const uint8_t *, size_t) [ChannelProxy = ccf::NodeToNode]
2025-10-15T17:05:17.359307Z 0 [info ] CCF/src/node/channels.h:1094 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] cannot receive encrypted message: not established a receive key, status=INITIATED
2025-10-15T17:05:17.359355Z 0 [fail ] CCF/src/node/rpc/forwarder.h:540 | Exception in void ccf::Forwarder<ccf::NodeToNode>::recv_message(const ccf::NodeId &, const uint8_t *, size_t) [ChannelProxy = ccf::NodeToNode]
2025-10-15T17:05:17.360406Z 0 [info ] CCF/src/node/channels.h:1094 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] cannot receive encrypted message: not established a receive key, status=INITIATED
2025-10-15T17:05:17.360460Z 0 [fail ] CCF/src/node/rpc/forwarder.h:540 | Exception in void ccf::Forwarder<ccf::NodeToNode>::recv_message(const ccf::NodeId &, const uint8_t *, size_t) [ChannelProxy = ccf::NodeToNode]
2025-10-15T17:05:17.370534Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:17.517292Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:17.517356Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:17.527142Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:17.761275Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:17.761334Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:17.770886Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:17.977864Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:17.977920Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:17.987761Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:18.177376Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:18.177445Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:18.187443Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:18.389280Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:18.389344Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:18.398761Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:18.637252Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:18.637313Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:18.646911Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:18.880531Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:18.880596Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:18.889424Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:19.131007Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:19.131066Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:19.140567Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:19.373310Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:19.373406Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:19.383063Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:19.581902Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:19.581956Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:19.591245Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:19.783544Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:19.783600Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:19.792740Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:20.009348Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:20.009404Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:20.018931Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:20.249303Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:20.249368Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:20.259662Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:20.441283Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:20.441339Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:20.451051Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:20.681270Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:20.681392Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:20.690797Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:20.925233Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:20.925286Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:20.934843Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:21.165278Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:21.165341Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:21.174966Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:21.385280Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:21.385336Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:21.395563Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:22.058104Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:22.058220Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:22.068081Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:22.758613Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:22.758703Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:22.768193Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:23.458460Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:23.458533Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:23.468123Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:24.158359Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:24.158441Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:24.168329Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:24.860131Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:24.860212Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:24.869672Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:25.560502Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:25.560580Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:25.570017Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:26.262009Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:26.262098Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:26.271605Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
2025-10-15T17:05:26.965057Z 0 [info ] CCF/src/node/channels.h:864 | Resetting channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da]
2025-10-15T17:05:26.965133Z 0 [info ] CCF/src/node/channels.h:841 | Initiating node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da].
2025-10-15T17:05:26.976940Z 0 [info ] CCF/src/node/channels.h:808 | Node channel with n[347df5d3c246f93f1a041ee2a5d79368ed0beffa81a8d45dcd5702d38cced1da] is now established.
In https://github.com/microsoft/CCF/actions/runs/18535993640
There were very significant I/O pauses preceding the issue, in other nodes as well, but that doesn't explain the verification failure followed by multiple successful establishments without any replication.