Pod not visible in Ochopod but running fine in Marathon

[pferrot]

I just experienced a situation where a pod was running fine in Marathon but it is not visible in Ochopod (disconnected from Zookeeper).

2016-06-21 08:04:39,174 - DEBUG - coordinator : zk state change -> "SUSPENDED" (connected)
2016-06-21 08:04:54,063 - DEBUG - watcher (marathon.ochothon) : exception trapped -> (../handlers/utils.py (78) -> ConnectionLoss ())
2016-06-21 08:04:55,484 - WARNING - coordinator : lost connection (SUSPENDED) / forcing a reset
2016-06-21 08:04:57,743 - DEBUG - coordinator : zk state change -> "LOST" (disconnected)
2016-06-21 08:04:59,263 - DEBUG - model (reactive) : watcher failure, terminating
2016-06-21 08:04:58,706 - DEBUG - watcher (marathon.ochothon) : reset ()

It seems that there is some logic to try and handle ZK connection issues, but in my case, the pod is still kind of half-working. Concretely, it was the Ochothon pod/cluster:

• I could connect to Ochothon and run regular commands (grep,...)
• But the Ochothon cluster itself was not visible when doing 'grep'
• Other cluster that depend on Ochothon would not see it