Investigating bigger spikes in multi-tiled playout latency

[jackjansen]

@rbouqueau assigning this to you primarily for discussion/insights.

In the (C#, Unity, VR2Gather) runs I am seeing occasional random spikes in playout latency of around 300ms.

This is a pipeline with 4 tiles at 2 qualities each (but no quality switching, so each tile receiver remains getting the same stream).

This is with sender and receiver running on Windows, lldash-relay running on Linux, on a "remote" network (I put "remote" between quotes,
because the three machines are actually on the same desk physically, but the connection between sender/receiver and relay goes through the firewall to the lldash-relay running on an external network port).

Here are two graphs of sample runs.

Run 1:

Run 2:

The spikes are at t=33 in run 1 and t=40 in run 2.

I investigated the raw timing statistics and in both cases one of my DASH receivers experienced a network latency that was 300ms larger than the normal network latency.

Google suggests that the minimum TCP RTO timeout on Windows is 300ms.

Could it be that what is happening here is a retransmission? How could I test that theory?

Edit: why was I thinking this is a retransmission? It is much more logical if it was a delay in the opening of the next segment... Is there some way I can detect at the higher (C#, Python) layer that we are at a segment boundary? Is this something that we could encode in the DashFrameMetaData?

[jackjansen]

I just checked whether the Python psutil package, which we use for gathering things like CPU/Memory/network usage during our runs (and which we also visualize) can give statistics on retransmissions, but apparently not.

The netstat command on Windows does give a total number of retransmissions, but I think this is retransmissions done by this computer, not the retransmissions requested.

[jackjansen]

There is a Windows API call that allows us to get statistics for a TCP socket: https://learn.microsoft.com/en-us/windows/win32/winsock/sio-tcp-info

That will give us the following structure: https://learn.microsoft.com/en-us/windows/win32/api/mstcpip/ns-mstcpip-tcp_info_v0

But: I'm not sure this is useful, and moreover it'll require major surgery to get access to the socket which is buried deep inside the signals layers....