Replica lag throttler not respecting lag time

[colin-strong]

Ad platform has configured LHMs to use the replica lag throttler.

Running an LHM shows the following logs:

INFO -- : Max current replica lag: 0

(Splunk)

However, Observe shows that replication lag for the database undergoing the LHM is over 40 minutes:

[tgwizard]

When I've seen this before it's because the LHM replica lag throttler resolves the addresses of the replicas by querying the writer:

https://github.com/Shopify/lhm/blob/5f748d2833883a9ccb74732e746a8bb2867ee83a/lib/lhm/throttler/replica_lag.rb#L89C7-L89C31

And then tries to connect to the replicas through those (IP) addresses, and fails because it doesn't have access (network / permissions, etc.). Usually replica access goes not directly to the DB but through ProxySQL, but these replica addresses that the writer sees are the IP addresses of the actual replica DBs.