New MPTCP connections won't be created if there's no connectivity on primary subflow

[arinc9]

Pre-requisites

• A similar issue has not been reported before.
• mptcp.dev website does not cover my case.
• An up-to-date kernel is being used.
• This case is not fixed with the latest stable (or LTS) version listed on kernel.org

What did you do?

Client device setup: Client device runs a proxy programme. The TCP traffic of the computer behind the client device is proxied to 102.132.201.141 using MPTCP.

$ uname -r
6.6.96

$ ip r
default via 192.168.65.1 dev eth1 proto static src 192.168.65.161 metric 1 
default dev qmimux0 proto static scope link src 100.127.48.95 metric 3 
[...]

$ ip mp e
192.168.65.161 id 1 subflow dev eth1 
100.127.48.95 id 2 subflow dev qmimux0 

Running iperf3 on a computer behind the client device:

iperf3 -c 102.132.201.141 --bind-dev enp7s0 -Z

MPTCP monitor running on client device: Working as expected.

$ ip mp m
[       CREATED] token=64f6b119 remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=34244 dport=31001
[   ESTABLISHED] token=64f6b119 remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=34244 dport=31001
[       CREATED] token=af201174 remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=34246 dport=31001
[   ESTABLISHED] token=af201174 remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=34246 dport=31001
[SF_ESTABLISHED] token=af201174 remid=0 locid=2 saddr4=100.127.48.95 daddr4=102.132.201.141 sport=45907 dport=31001 backup=0 ifindex=19
[SF_ESTABLISHED] token=64f6b119 remid=0 locid=2 saddr4=100.127.48.95 daddr4=102.132.201.141 sport=48649 dport=31001 backup=0 ifindex=19
[        CLOSED] token=64f6b119
[        CLOSED] token=af201174

Add this iptables rule on client device to simulate no connectivity on the second subflow:

iptables -t mangle -A PREROUTING -i qmimux0 -j DROP

MPTCP monitor running on client device: Expected operation, but the second TCP stream won't be reported as closed.

$ ip mp m
[       CREATED] token=0fa4dc43 remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=35594 dport=31001
[   ESTABLISHED] token=0fa4dc43 remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=35594 dport=31001
[       CREATED] token=6eba6bed remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=35610 dport=31001
[   ESTABLISHED] token=6eba6bed remid=0 locid=0 saddr4=192.168.65.161 daddr4=102.132.201.141 sport=35610 dport=31001
[        CLOSED] token=0fa4dc43
(It won't report CLOSED for 6eba6bed)

Add this iptables rule on client device to simulate no connectivity on the primary subflow:

iptables -t mangle -D PREROUTING -i qmimux0 -j DROP
iptables -t mangle -A PREROUTING -i eth1 -j DROP

MPTCP monitor running on client device: No CREATED or ESTABLISHED messages. Only CLOSED is reported after the TCP connection is terminated by iperf3.

[        CLOSED] token=faab2c24

iperf3 log:

iperf3: error - unable to receive control message - port may not be available, the other side may have stopped running, etc.: Connection reset by peer

The route that is at the top of the routing table list becomes the primary subflow for MPTCP. So deleting it as an endpoint won't change the behaviour.

$ ip mp e
192.168.65.161 id 1 subflow dev eth1 
100.127.48.95 id 2 subflow dev qmimux0 
$ ip mp e d i 1
$ ip mp e
100.127.48.95 id 2 subflow dev qmimux0 

Excerpt: New MPTCP connections should still be created when there's no connectivity on the primary subflow but there's connectivity on the remaining subflows. This could be implemented in a way that, when creating an MPTCP connection, a different endpoint should be used if the previous one times out.

[matttbe]

Hi @arinc9,

Thank you for the bug report.

Excerpt: New MPTCP connections should still be created when there's no connectivity on the primary subflow but there's connectivity on the remaining subflows. This could be implemented in a way that, when creating an MPTCP connection, a different endpoint should be used if the previous one times out.

If I understood correctly, this is similar to #585, right?

For the connection establishment, MPTCP should respect the routing rules set by the userspace. If the route is unavailable, that's not MPTCP's responsibility. If we stop respecting the routing decisions, that will probably lead to complex code, and a confusing (and "illegal"?) behaviour. Maybe what you need is something like mwan3?

[arinc9]

Hey Matt.

Yes. It's the same issue, I created this issue to precisely convey what the issue is and what I'm looking for to address it.

mwan3 is just too complex and I don't think it makes sense to use it for this specific behaviour of the MPTCP implementation. I already have a simple userspace daemon that modifies the routing table to mitigate this issue. However, I'd still like to have this dealt with on the MPTCP implementation.

This is how the MPTCP implementation currently works, correct me if I'm wrong:

Use the primary route as an endpoint to create an MPTCP connection. In case of timeout, fail.

This is how I'd like it to work:

Use the primary route as an endpoint to create an MPTCP connection. In case of timeout, check if there are more endpoints on the MPTCP endpoint table. If not, fail. If there is, use the next endpoint on the MPTCP endpoint table to create the connection. Repeat until it tries creating the connection using every endpoint. In case of timeout and no more endpoints, fail.

I think this enhancement would fall within the purpose of MPTCP's in-kernel path management.

Do you agree? Do you see a way to implement this on the path manager or somewhere else?

[matttbe]

mwan3 is just too complex and I don't think it makes sense to use it for this specific behaviour of the MPTCP implementation. I already have a simple userspace daemon that modifies the routing table to mitigate this issue. However, I'd still like to have this dealt with on the MPTCP implementation.

This is not specific to MPTCP: MPTCP respects the routing rules like any L4 protocol would do. If MPTCP is not able to initiate a connection because there is an issue on the route configured by the userspace, all other protocols using this route will have the same issue.

This is how the MPTCP implementation currently works, correct me if I'm wrong:

Use the primary route as an endpoint to create an MPTCP connection. In case of timeout, fail.

Correct. When initiating the MPTCP connection, the kernel will resend the SYN+MP_CAPABLE. After a few attempts (controlled by the net.mptcp.syn_retrans_before_tcp_fallback sysctl knob), it will send SYNs without MP_CAPABLE. Note that by default, it will retry for a long time (128?), but the userspace app will certainly timeout before.

This is how I'd like it to work:

Use the primary route as an endpoint to create an MPTCP connection. In case of timeout, check if there are more endpoints on the MPTCP endpoint table. If not, fail. If there is, use the next endpoint on the MPTCP endpoint table to create the connection. Repeat until it tries creating the connection using every endpoint. In case of timeout and no more endpoints, fail.

I think this enhancement would fall within the purpose of MPTCP's in-kernel path management.

The MPTCP path-manager is in fact not responsible for creating the initial path. It is only involved when the connection has already been created.

Do you agree? Do you see a way to implement this on the path manager or somewhere else?

I understand that with MPTCP, multiple paths can be used for the same connection, and then it might make to let the kernel picking another route, but I don't think that's a good idea:

• Routing rules are controlled by the userspace, and applications can influence them, e.g. with setsockopt(SO_BINDTODEVICE). MPTCP should not bypass that. The network layer is in charge of that.
• If MPTCP was really in charge of that, to which endpoint should it fallback to? In which order? Switch between v4 and v6 allowed? After what timeout and/or errors? What if the app has a short timeout not allowing a fallback to different interfaces? It seems difficult to handle that on the kernel side.

Handling this on the userspace side should not be too difficult: mwan3 (or any other daemons) has to monitor a path (ping, TCP connection, netlink events, etc.), and change the default route when needed.

while true; do
  if ! timeout 1 ping -I${IFACE} -qc1 ${DEST} >/dev/null; then
      <change default route>  ## TODO: more tolerant with timeout (bufferbloat) and temp issues?
  fi
  sleep 1
done

Or the app should implement a Happy Eyeballs technique to check multiple routes, similar to what is already done to pick a v4 or v6 address (or pick between HTTP, HTTPS, QUIC, etc.)

I understand it is not the answer you are expecting, but I don't think MPTCP should bypass take care of a L3 issue.

[arinc9]

Good points. I've already made this script that handles it well. Regarding MPTCP handling it; even if we dealt with everything else, an acceptable timeout for creating a connection would be much more than 900ms (30 seconds?). And that's too long for me.

https://github.com/bondingshouldbefree/rutos-sdk/blob/b52456781e765294d7aaff84561629735970b6be/files/usr/local/usr/sbin/xpedite-route