Firewall getting hammered by STUN requests to/from random UDP ports

[brianjmurrell]

Hi. I have a Homeassistant (HA) installation using go2rtc for cameras. This HA is exposed to the Internet with SSL but behind a firewall with the main HA port opened for access. All of that is working just fine.

When I connect to the HA instance from the HA Android app while on the Internet, everything in the app works. I see camera feeds, etc., but while I am connected from my Android phone, while on the Internet (and everything is working fine), the firewall that this HA is behind is logging a massive amount of UDP connections on random ports (>~33000) being sent between the Android phone and HA. Wireshark is telling me these are STUN requests.

Beyond those STUN requests on random UDP ports I do also see HA making (a different kind of) STUN requests to a few AWS hosts on ports 80 and 3478. These requests do get responses, describing the NAT situation that HA is behind

ec2-23-21-77-118.compute-1.amazonaws.com	62e48b77f6b84952809968f5188afb55.local	STUN	Binding Success Response XOR-MAPPED-ADDRESS: [redacted]:39166 MAPPED-ADDRESS: [redacted]:39166 RESPONSE-ORIGIN: 34.237.188.15:3478

I suspect these requests and responses are what are actually making the camera feeds functional between HA and the Android phone.

The barrage of the other kinds of STUN requests (none of which ever get any responses) from the Android phone to the HA address look like:

Frame 491274: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits) on interface gw:eth0.2, id 1
Ethernet II, Src: 4c:42:1e:3b:d0:19, Dst: 6c:b0:ce:f5:1e:4b
Internet Protocol Version 4, Src: [Android HA app address redacted] ([Android HA app address redacted]), Dst: [HA's public IP address redacted] ([HA's public IP address redacted])
User Datagram Protocol, Src Port: 63389 (63389), Dst Port: 44884 (44884)
Session Traversal Utilities for NAT
    Message Type: 0x0001 (Binding Request)
        .... ...0 ...0 .... = Message Class: 0x00 Request (0)
        ..00 000. 000. 0001 = Message Method: 0x0001 Binding (0x001)
        ..0. .... .... .... = Message Method Assignment: IETF Review (0x0)
    Message Length: 92
    Message Cookie: 2112a442
    Message Transaction ID: 666c6475356f414f4955664b
    [STUN Network Version: RFC-5389/8489 (3)]
    Attributes
        USERNAME: EsjgLMmhzXQJPWbP:mfPZ
            Attribute Type: USERNAME
                0... .... .... .... = Attribute Type Comprehension: Required (0x0)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 21
            Username: EsjgLMmhzXQJPWbP:mfPZ
            Padding: 3
        GOOG-NETWORK-INFO
            Attribute Type: GOOG-NETWORK-INFO
                1... .... .... .... = Attribute Type Comprehension: Optional (0x1)
                .1.. .... .... .... = Attribute Type Assignment: Designated Expert (0x1)
            Attribute Length: 4
            Google Network ID: 0
            Google Network Cost: Max (999)
        ICE-CONTROLLING
            Attribute Type: ICE-CONTROLLING
                1... .... .... .... = Attribute Type Comprehension: Optional (0x1)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 8
            Tie breaker: 68c0d0c5f29a0537
        USE-CANDIDATE
            Attribute Type: USE-CANDIDATE
                0... .... .... .... = Attribute Type Comprehension: Required (0x0)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 0
        PRIORITY
            Attribute Type: PRIORITY
                0... .... .... .... = Attribute Type Comprehension: Required (0x0)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 4
            Priority: 1845501695
        MESSAGE-INTEGRITY
            Attribute Type: MESSAGE-INTEGRITY
                0... .... .... .... = Attribute Type Comprehension: Required (0x0)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 20
            HMAC-SHA1: 038ed2014e8a5f4b6bf25abe456fcbd5bbf3522f
        FINGERPRINT
            Attribute Type: FINGERPRINT
                1... .... .... .... = Attribute Type Comprehension: Optional (0x1)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 4
            CRC-32: 0xc707649b [correct]
            [CRC-32 Status: Good]

And similarly, HA also sends such unresponded requests:

Frame 481795: 142 bytes on wire (1136 bits), 142 bytes captured (1136 bits) on interface gw:eth0.2, id 1
Ethernet II, Src: 6c:b0:ce:f5:1e:4b, Dst: 4c:42:1e:3b:d0:19
Internet Protocol Version 4, Src: [HA's public IP address redacted] ([HA's public IP address redacted]), Dst: [Android HA app address redacted] ([Android HA app address redacted])
User Datagram Protocol, Src Port: 43173 (43173), Dst Port: 36642 (36642)
Session Traversal Utilities for NAT
    Message Type: 0x0001 (Binding Request)
        .... ...0 ...0 .... = Message Class: 0x00 Request (0)
        ..00 000. 000. 0001 = Message Method: 0x0001 Binding (0x001)
        ..0. .... .... .... = Message Method Assignment: IETF Review (0x0)
    Message Length: 80
    Message Cookie: 2112a442
    Message Transaction ID: dabdfdfe74e9fef50b0a471b
    [STUN Network Version: RFC-5389/8489 (3)]
    Attributes
        USERNAME: WVuw:mBgyoUTKOzXuOPks
            Attribute Type: USERNAME
                0... .... .... .... = Attribute Type Comprehension: Required (0x0)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 21
            Username: WVuw:mBgyoUTKOzXuOPks
            Padding: 3
        ICE-CONTROLLED
            Attribute Type: ICE-CONTROLLED
                1... .... .... .... = Attribute Type Comprehension: Optional (0x1)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 8
            Tie breaker: 806fff457b712d6b
        PRIORITY
            Attribute Type: PRIORITY
                0... .... .... .... = Attribute Type Comprehension: Required (0x0)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 4
            Priority: 1694498815
        MESSAGE-INTEGRITY
            Attribute Type: MESSAGE-INTEGRITY
                0... .... .... .... = Attribute Type Comprehension: Required (0x0)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 20
            HMAC-SHA1: 88af451e1687eb9f2206f60901f79892733a5042
        FINGERPRINT
            Attribute Type: FINGERPRINT
                1... .... .... .... = Attribute Type Comprehension: Optional (0x1)
                .0.. .... .... .... = Attribute Type Assignment: IETF Review (0x0)
            Attribute Length: 4
            CRC-32: 0xf823590d [correct]
            [CRC-32 Status: Good]

Clearly given that neither of these are being responded to nor permitted through the firewall, they are no necessary. They are just leading to a lot of noise in the firewall logs as well as blocking of the remote/mobile users due to their activity looking like they are aggressively trying to attack the network behind the firewall.

Are these particular STUN probes necessary and if not, can they be disabled?

[brianjmurrell]

I'm actually just noticing that one of my camera feeds on the Android client only says RTC on the bottom left when I am connected by the local LAN. When connected via the Internet (i.e. with the firewall between the client and HA instance) the client camera card actually says MSE on the bottom left.

So I guess I am not actually working "perfectly fine" when connected through the firewall. :-( I suppose I need to do something more/else to get RTC working through the firewall, yes? Is this scenario documented somewhere?

[brianjmurrell]

OK. Been doing a bit of reading. I guess all of these (unreplied) STUN requests are an attempt at UDP hole punching where one or both sides of the connection are attempting to get IP:port pairs into their connection state tables. But clearly that is failing since this STUN hammering seems to continue on and on. Additionally, examining the IP:port pairs in a request going in direction does not result in a reciprocal request going in the other direction which I would expect is needed in order for both sides to get the respective connections in their state tables.

I have taken note of AlexxIT/WebRTC#378 and the documentation is refers to at https://github.com/AlexxIT/go2rtc#module-webrtc. Accordingly I have tried adding both (not at the same time of course):

webrtc:
  candidates:
    - my.external.ip.address:8555  # if you have a static public IP address

and:

webrtc:
  candidates:
    - stun:8555  # if you have a dynamic public IP address

to my configuration.yaml file but I never see any connection attempt to that port on my firewall.

I guess if I assume a worst-case scenario where both my HA (connected through a Linux netfilter SNAT/firewall -- is that symmetric NAT?) and mobile phone are connected through symmetric NATs, so that neither side can predict which external port (or perhaps even IP address) they will be assigned on a new connection, there is no way for them both to send packets to their reciprocal ports to create the necessary state table entries to allow them to communicate. Phooey.

Can't NAT (and IPv4) just die already? 😀 I say that, but my HA server does have an IPv6 address and my mobile client also has IPv6 (and is using CGNAT for IPv4 connectivity). My understanding was that dual-stack implementations such as Android employs are supposed to prefer IPv6 if a host has both IPv4 and IPv6 addresses. I wonder why this device is not doing so.

I guess where I am still confused here is where port 8555 comes into play in all of this.

[brianjmurrell]

Interesting progress.

Sadly, even though my phone gets an IPv6 address on my provider's network, the totally lame provider doesn't actually support IPv6 and so my phone falls back to IPv4, (likely with symmetric CGNAT).

So I switched my phone to a different mobile provider that does properly support IPv6.

The interesting changes that resulted in were that my phone was now using IPv6 to connect to HA and the STUN Binding Requests being made from my phone to HA through the firewall resulted in successful Binding Responses from HA. The result was achieving use of an out-of-band a UDP session presumably for the video stream?

My camera card in HA was still showing MSE and I also now saw connection requests to port 18555 (not 8555) being blocked as I have not yet installed the forwarding rule in the firewall.

But, upon installing that forwarding rule, I now have RTC showing in my camera card. Yay! Interestingly enough though my tcpdump (Wireshark) shows nothing in port 18555 nor does the conntrack table on the firewall, nor does the netfilter rule counters. So I'm a bit puzzled about that.

I might have to have a serious thought about switching back from the mobile provider that only pretends to support IPv6 to the one that actually does.

I see that TURN is an option. Is that a solution for the double-ended symmetric NAT? Does it only work on a VPS or can a TURN server work on the firewall device that is doing the symmetric NAT for the webrtc/go2rtc?

[AlexxIT]

Can't NAT (and IPv4) just die already? 😀

This is was really funny. I've never had any experience with a working IPv6 in my entire life. Useless experiments on a local network do not count.

[AlexxIT]

Read this. Perhaps you'll have fewer questions:
https://github.com/AlexxIT/go2rtc/blob/master/internal/webrtc/README.md

[brianjmurrell]

Read this. Perhaps you'll have fewer questions:

Some notes from that README.md:

Usually, WebRTC uses public STUN servers to establish a connection outside LAN, such servers are only needed to establish a connection and are not involved in data transfer

But even STUN can't help hole punching when both ends are behind symmetric NAT, correct?

• Uses UDP hole punching technology to bypass NAT even if you not open your server to the World
• For about 20% of users, the techology will not work because of the Symmetric NAT

These 20% are the people with both ends behind symmetric NAT (i.e. CGNAT on one end and Linux' netfilter at the other)? And for these 20% of people, they are just screwed with no solution? Maybe TURN is a solution, if they have or can stand up a TURN server somewhere.

By default go2rtc uses fixed TCP port and fixed UDP ports for each direct WebRTC connection - listen: ":8555".

This is interesting. Am I understanding this to mean that by default the random UDP ports behaviour that I am seeing here is not the default that that by default the client will try to create this peer-to-peer connections with the go2rtc (i.e. in HA) server using it's fixed port of 8555? Because clearly that is not what I am seeing here. But comment in the configuration example above it seems to imply something different:

    # range for random UDP ports [min, max] to be used for connection
    # not related to the `listen` option
    udp_ports: [ 50000, 50100 ]

So the random UDP ports connections are separate from the usage of the 8555 port?

What seems very interesting is that when using IPv4 on (what I suspect is) dual ended symmetric NAT, I don't ever see any connection on port 8555 at all. I do see that port being used for IPv6, where there is direct peer-to-peer connectivity without any NAT. Is that possible for any reason?

I suppose one option for IPv4 is to open a smallish range of UDP ports and forward them to HA and set that range using the aforementioned udp_ports: [ 50000, 50100 ] option.

This is was really funny. I've never had any experience with a working IPv6 in my entire life. Useless experiments on a local network do not count.

I have a fully functional IPv6 LAN along with two upstream IPv6 Internet connections. One of those is an HE tunnel which I've had for about 13 years.

HE tunnels are a great way to get your feet wet if you cannot get IPv6 from an ISP.

Assuming you have an interest that is. It's not sounding like you do.

But it's really liberating and enjoyable to get away from the constant headaches of having to try to deploy more than the most simple of network protocols on IPv4 NAT.

[AlexxIT]

WebRTC uses 3 types of connection simultaniously:

• direct connection - direct IP + fixed or random port
• STUN connection - public IP + random port
• TURN connection - fixed IP and port of public TURN server

listen and candidates uses for direct connection