fix: multistream-select negotiation on outbound substream over webrtc #465

haikoschol · 2025-11-05T13:39:36Z

This PR changes how multistream-select messages received on an opening outbound substream are handled on webrtc connections. For details please see the issue description.

The PR builds on top of #441 for the str0m update. But looking at it again I realized that it also made changes to multistream-select handling. I'll look into what those are and how they relate to my changes. In any case, I believe my changes bring the implementation closer to compliance with the multistream-select spec and enable interoperability with smoldot.

fixes #464

…cation, modify multistream protocol message, modify opening based on str0m api changes

haikoschol · 2025-11-05T13:46:59Z

src/transport/webrtc/connection.rs

            protocol: protocol.to_string(),
        });

+        // self.rtc.channel(channel_id).unwrap().set_buffered_amount_low_threshold(1024);


This is an unrelated issue.

I've removed this line since it is unrelated to this PR and should be address later when working on backpressure.

haikoschol · 2025-11-05T13:51:44Z

src/transport/webrtc/connection.rs

                event = self.handles.next() => match event {
                    None => unreachable!(),
-                    Some((channel_id, None | Some(SubstreamEvent::Close))) => {
+                    Some((_, None)) => {}


This is a superficial fix for another unrelated issue which I haven't investigated yet. Without this change, a "channel closed" event is fired repeatedly after a while. I believe this is caused by SubstreamHandle::tx being dropped, presumably when the substream(/channel?) is closed. The fact that the event continues firing suggests that SubstreamHandle::rx continues to be polled. Maybe the handle simply needs to be removed from self.handles?

I've also seen this happen when briefly testing with a rust-libp2p dialer, so this is not related to smoldot interop.

I just tested again without this change and now I don't get repeating "channel closed" messages in the logs. Presumably this is due to a change somewhere else that I can't pin down right now. In any case, it was unrelated to multistream-select negotiation.

lexnv · 2025-11-12T09:22:05Z

src/transport/webrtc/connection.rs

-
-                        self.rtc.direct_api().close_data_channel(channel_id);
                        self.channels.insert(channel_id, ChannelState::Closing);
-                        self.handles.remove(&channel_id);


Wouldn't this lead to memory leaks due to keeping the channel ids around? 🤔

You are right. I don't remember what exactly I had in mind with that change. Probably expecting that Event::ChannelClose would be emitted eventually and Connection::on_channel_closed() called, which does the full cleanup. But that's not exactly how it works.

The way it was before is incorrect as well, but that's out of scope for this PR since it is related with signaling of substream closure via the WebRTC protobuf frame.
Anyway, I'll revert this change and we'll revisit it in a follow-up PR.

This moves a workaround for fixing multistream-select negotiation over WebRTC transport to WebRTC specific code in order to avoid introducing interop failures with previous versions of litep2p when using other transports.

haikoschol · 2025-11-13T06:00:59Z

@lexnv I've made some more changes and think this PR is now in a pretty good state.

removed a few unrelated changes
merged master
implemented the third option discussed in webrtc: update str0m, fix multistream message decoding and encoding #441
moved the fix to outbound substream negotiation from WebRtcConnection::on_outbound_opening_channel_data() to WebRtcDialerState::register_response() in order to fix some of its tests
handled the remaining negotiation response shapes in WebRtcDialerState::register_response() that are covered by the tests

I had to change a few tests, but hopefully only their implementation, not their meaning.

If I haven't missed anything, all changes should now be limited to the webrtc-specific code.

I've tested the following with a litep2p listener based on this PR:

With a smoldot-ish dialer running in Chrome:

connection establishment
multistream-select negotiation on outbound substreams
multistream-select negotiation on inbound substreams
outbound ping
inbound ping

With a rust-libp2p dialer using webrtc.rs 0.13.0:

connection establishment
multistream-select negotiation on inbound substreams
perf behaviour

src/multistream_select/dialer_select.rs

lexnv · 2025-11-13T09:30:11Z

src/multistream_select/dialer_select.rs

+            map_err(|_| error::NegotiationError::ParseError(ParseError::InvalidData))?;
+
+        let payload = tail[..len].to_vec();
+


nit: It would be good if we can greak the code such that we provide better trace logging:

let message = Message::decode(payload.into()); tracing::trace!( target: LOG_TARGET, ?message, "Decoded message while registering response", ); let mut protocols = match message { .. };

Let me know if this makes sense or if you think it will polute too much the logs

I think this is fine, especially as a trace.

lexnv · 2025-11-13T09:35:16Z

src/multistream_select/dialer_select.rs

+            let (len, tail) = unsigned_varint::decode::usize(remaining).
+                map_err(|_| error::NegotiationError::ParseError(ParseError::InvalidData))?;


Does this mean we are expecting multiple [ len_prefix ++ Message ] to arrive from the register_response of the WebRTC?

Who would normally exhibit this behavior (libp2p or smoldot)?

src/multistream_select/dialer_select.rs

lexnv · 2025-11-13T09:45:25Z

src/multistream_select/dialer_select.rs

+        bytes.put_u8((proto.as_ref().len() + 1) as u8); // + 1 for \n
+        let _ = Message::Protocol(proto).encode(&mut bytes).unwrap();
+
+        let expected_message = bytes.freeze().to_vec();


dq: This slightly changes the meaning of the messages.

Before we were expecting:

[Protocols(/multistream/1.0.0), Protocol(/13371338/proto/1)]

Now we are expecting:

// \n added [Header(/multistream/1.0.0 /\n), // \n counted but not checked Protocol(/13371338/proto/1 [])

Hmm, one of those representation can't be spec compliant right?

Also are we ignoring malformated messages:

we decode the frame size correctly

but we don't check then if the last character is \n or not sufficeint bytes provided?

Right now the test ensures that the wire format that WebRtcDialerState::propose() produces is what we expect. Before it actually ensured that whatever propose() produces can be decoded with Message::decode(). So I did change the semantics of the test after all. But I would argue that the new meaning is more useful. We could also add decoding and assert we get the expected strings back, but due to the changes made, this can no longer be done by just calling Message::decode().

lexnv

Great job so far! Feels like we are getting closer to figuring out the multistream select 🙏

My questions revolve around the spec correctness of the implementation, or if we basically implemented a tiny subset in the past? And mainly, which implementation is causing us to extensively parse the frames (libp2p or smoldot or both?) 🤔

- add logging calls - clean up imports - add constant for Protocol::try_from(&b"/multistream/1.0.0"[..])

src/multistream_select/dialer_select.rs

haikoschol · 2025-11-13T12:00:59Z

And mainly, which implementation is causing us to extensively parse the frames (libp2p or smoldot or both?) 🤔

The litep2p implementation is causing us to extensively parse the frames. :D This is because we always deal with the entire content of a WebRTC protobuf frame, which may contain one or more multistream messages.

In rust-libp2p the WebRTC implementation uses MessageIO and LengthDelimited in a layer below the multistream-select logic. The multistream-select logic gets fed from a stream that yields one multistream_select::protocol::Message at a time, even if the header and the selected protocol are in the same WebRTC protobuf frame.

I hope this makes some sense. I'll go into detail in our call today.

lexnv · 2025-11-18T12:49:15Z

src/multistream_select/dialer_select.rs

+            })?;
+
+        if len > tail.len() {
+            break;


nit: I think this would miss a few protocols, maybe return an error and add a debug log?

I just tried this and it caused the test header_line_missing() to fail. This is because that test used Message::Protocols to encode two protocol names (without a header) to use as a negotiation response. This isn't spec compliant in two ways that are irrelevant to what is being tested, so I changed the test to send one protocol name.

lexnv

The logic looks good to me!

Great job on getting str0m updated and figuring out the multistream select implementation 🙏

timwu20 and others added 5 commits November 5, 2025 17:47

update strom dependency which fixes disable remote fingerprint verifi…

196d625

…cation, modify multistream protocol message, modify opening based on str0m api changes

change str0m to 0.11.1

b875bbd

fix lint

925caa9

set and listen on ChannelBufferedAmountLow after stream closure

4200246

fix: multistream-select negotiation for outbound webrtc substreams

85fae21

haikoschol force-pushed the haiko-webrtc-outbound-multistream-nego-fix branch from 93e6151 to 85fae21 Compare November 5, 2025 13:43

haikoschol commented Nov 5, 2025

View reviewed changes

This was referenced Nov 5, 2025

Litep2p Server <-> Smoldot Client Integration ChainSafe/gossamer-parity#75

Open

multistream-select implementation used by WebRTC doesn't conform to the spec #75

Open

lexnv mentioned this pull request Nov 12, 2025

webrtc: update str0m, fix multistream message decoding and encoding #441

Closed

lexnv reviewed Nov 12, 2025

View reviewed changes

This was referenced Nov 12, 2025

[litep2p-milestone]: Implement WebRTC transport layer #312

Open

feat: support WebRTC transport for libp2p client and server lexnv/litep2p-perf#2

Open

haikoschol added 6 commits November 12, 2025 21:19

close data channel immediately when substream is closed

2ec539b

remove setting of buffered amount low threshold in str0m

4e4a2d3

remove special case of receiving None instead of a SubstreamEvent

7947bf6

move handling of trailing newline to webrtc_listener_negotiate()

3765d52

This moves a workaround for fixing multistream-select negotiation over WebRTC transport to WebRTC specific code in order to avoid introducing interop failures with previous versions of litep2p when using other transports.

cover all negotiation responses included in tests

fdedc4f

Merge branch 'master' into haiko-webrtc-outbound-multistream-nego-fix

1cd945b

lexnv reviewed Nov 13, 2025

View reviewed changes

src/multistream_select/dialer_select.rs Outdated Show resolved Hide resolved

lexnv reviewed Nov 13, 2025

View reviewed changes

src/multistream_select/dialer_select.rs Outdated Show resolved Hide resolved

lexnv reviewed Nov 13, 2025

View reviewed changes

src/multistream_select/dialer_select.rs Outdated Show resolved Hide resolved

lexnv reviewed Nov 13, 2025

View reviewed changes

src/multistream_select/dialer_select.rs Outdated Show resolved Hide resolved

lexnv reviewed Nov 13, 2025

View reviewed changes

src/multistream_select/dialer_select.rs Outdated Show resolved Hide resolved

lexnv reviewed Nov 13, 2025

View reviewed changes

haikoschol added 2 commits November 13, 2025 18:28

address review feedback

6dcb15d

- add logging calls - clean up imports - add constant for Protocol::try_from(&b"/multistream/1.0.0"[..])

extract loop to drain_trailing_protocols()

9621e3c

lexnv reviewed Nov 13, 2025

View reviewed changes

src/multistream_select/dialer_select.rs Outdated Show resolved Hide resolved

lexnv reviewed Nov 18, 2025

View reviewed changes

lexnv requested a review from dmitry-markin November 18, 2025 12:50

lexnv approved these changes Nov 18, 2025

View reviewed changes

haikoschol added 3 commits November 19, 2025 14:29

fmt

daa8f71

Merge branch 'master' into haiko-webrtc-outbound-multistream-nego-fix

afd44c8

consider truncated multistream messages invalid data

99f6547

		map_err(\|_\| error::NegotiationError::ParseError(ParseError::InvalidData))?;

		let payload = tail[..len].to_vec();

		let (len, tail) = unsigned_varint::decode::usize(remaining).
		map_err(\|_\| error::NegotiationError::ParseError(ParseError::InvalidData))?;

fix: multistream-select negotiation on outbound substream over webrtc #465

Are you sure you want to change the base?

fix: multistream-select negotiation on outbound substream over webrtc #465

Uh oh!

Conversation

haikoschol commented Nov 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haikoschol commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haikoschol Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lexnv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

haikoschol commented Nov 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haikoschol Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lexnv left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

haikoschol commented Nov 13, 2025 •

edited

Loading

haikoschol Nov 13, 2025 •

edited

Loading

haikoschol Nov 19, 2025 •

edited

Loading