Link peer unit testing challenge -- timing

[jamshark70]

Question rather than an issue --

I'm informally responsible for unit tests of SuperCollider's LinkClock. Most of the tests are fine.

We have one test called test_newLinkClock_shouldNotChangeLinkTempo.

1. Create a LinkClock at 150 bpm.
2. Wait 500 ms for the Link session to stabilize.
3. Create a second LinkClock at 60 bpm.
4. This should sync clock2's tempo (and phase etc., but tempo is what we're testing). On the SC side, this causes a \tempo notification to be broadcast, which signals the test to move ahead.
5. Then we check if clock1's tempo is still 150 bpm: true = pass.
6. Both clocks (peers) are then destroyed at the same time.

On my machine, if I run this test in isolation, it passes.

In our github CI, the test sporadically fails.

I can also reproduce the failure if I run the test multiple times in quick succession. When it fails, it looks like clock2 controls the Link session's tempo, despite being created later. But, if I increase the delay after creating the first clock, the failures disappear.

So there's a conflict between Link requiring some time to ensure that the session is created correctly vs the desire to minimize hardcoded delays in a unit testing suite that already takes longer to run than we might like.

So the question is how best to handle this.

Is there a recommendation for how long to wait for the first Link peer to stabilize the session, so that we can be sure of its state?

Or (better) does the Link API emit some signal when the session is ready to be trusted? Above, for instance, the second peer gets a tempo-change notification if, at connection time, its tempo disagrees with the session's established tempo. But the first peer, AFAICS, gets nothing. So, on our side, I don't know how to be sure that the clock has actually synced up with the session. Understood that, under normal use, you'd create the peer and probably start playing up to several minutes later -- but for testing purposes, it would really help not to have to hardcode long wait times.

Any advice welcome -- thanks.

[fgo-ableton]

Link does not provide any hooks to query the connection state or similar. In general using code that is asynchronous and does networking can be quite tricky on CI. Also, in case of Link, if the tests run in parallel on two CI nodes, they could discover each other and thus behave not as expected.
So I would recommend using a mock, and not the actual Link library in tests. Theoretically this should allow testing all code on the SC side. This is also what we do for our engine code.

[jamshark70]

Link does not provide any hooks to query the connection state or similar.

I don't think it needs to be a full query of the state -- but, it may be useful if, in addition to the tempo, start/stop and numpeers callbacks, there were a one-shot callback to fire when the link session is joined. If we had that callback, then, for instance, SC could do:

(
~makeLinkClock = { |action, latency(Server.default.latency)|
	var clock = LinkClock.new.latency_(latency);
	fork {
		var linked = false;
		var cond = CondVar.new;
		var resp = SimpleController(clock)
		.put(\joinedSession, {
			linked = true;
			resp.remove;
			cond.signalAll;
		});
		cond.waitFor(5) { linked };
		if(linked.not) {
			"LinkClock failed to join session".warn;
		};
	};
	clock
};
)

... to implement a basic timeout, for instance.

That's a good point about parallel CI runs -- thanks!

[fgo-ableton]

The session is not really something the user of the Link API should have to worry about. So I'd rather not expose anything related to that. Also, checking if Link calls the right callbacks after init is really something you should not have to worry about in SC land - that should be tested in Link I guess. 😉
However, in this particular case the numPeersCallback should do the job, right? The session of the peer that started first will be joined. Once both peers have discovered each other, tempo changes and such should propagate.

[jamshark70]

However, in this particular case the numPeersCallback should do the job, right?

It's not exactly the same thing.

(
l = LinkClock.new;
~lResp = SimpleController(l)
.put(\numPeers, { l.numPeers.debug("l peers") });
)

(
m = LinkClock.new;
~mResp = SimpleController(m)
.put(\numPeers, { m.numPeers.debug("m peers") });
)

Both clocks get the numPeers notification after the second one joins -- which makes sense, because the number of peers has changed for both of them.

What I'm asking about is, how does the first peer, the one to initialize the session, know that the session is fully up and running, before the second peer comes in? All I know is that it takes some variable number of milliseconds, ranging from a few to a few hundred. Above, l doesn't get any notification at all until m joins. Sure, I understand that in normal usage, there are likely to be at least a few seconds between peers joining, so it's generally OK to take it on faith that the first one initiated the session. I'm just pointing out that it's currently impossible to confirm that (though it is possible to confirm the presence of multiple peers). In a testing scenario, it may be useful to know that the session is ready for the second peer to join.

In any case, I think this issue can be closed -- keep it open if there is any chance of adding that callback, but I doubt that's going to happen. The above answers my questions about testing, so, OK to close.

[fgo-ableton]

Ok, I see.
A little background: The Link session is "running" as soon as the initializer returns. After calling enable it will search for network interfaces and send out its state. When a peer receives a state from another peer it will join the other peers session if it is "older" than its own session.
There is no state change happening until the second peer joins the first peers session. So we really can't add callback before this.
However, when enabling both peers directly after each other, determining what the "older session" is has some inaccuracies due to network jitter. Waiting 5 ms or so before enabling the second peer should make sure it always connects to the first peers session.

[jamshark70]

OK, understood.

Waiting 5 ms or so before enabling the second peer should make sure it always connects to the first peers session.

In that particular testing scenario, it was waiting 500 milliseconds, and the behavior was inconsistent. That's why I was confused. When actually using Link, I didn't remember it taking that long, but I also can't figure out why a sequence of steps "1/ create first clock; 2/ wait 500 ms; 3/ create second clock" sometimes follows the first clock and sometimes follows the second (but only when running the test multiple times in relatively rapid succession, like 100 ms between trials).

How long does it take for a Link session to disappear when SC deletes the last LinkClock?

[fgo-ableton]

How long does it take for a Link session to disappear when SC deletes the last LinkClock?

Usually a peer sends a bye message when it is disabled. If that message gets dropped on the network it times out after 6 seconds.