Fix sending compound messages corruption when including messages larger than 64KB

[pracucci]

Cortex uses memberlist with a custom transport implementation, based on TCP protocol. We have a use case to send large messages, sometimes bigger than 64KB.

When memberlist implementation builds a compound message, it doesn't take in account the message length. Since in the compound message the length of an inner message is stored as a uint16, the compound message gets corrupted if contains an inner message larger than 64KB.

To solve this issue, I propose to not include messages larger than 64KB in a compound message.

Notes:

• We verified this fix in Cortex and solves our use case.
• This PR is a follow up of the fix done in Fix for compound messages containing >255 messages. #239

[hashicorp-cla]

All committers have signed the CLA.

[alvinlin123]

@markan I wondering if you can take a look at this PR? I thought you might be a good candidate because you looked at the related PR: #239

Thanks in advance :)

[alvinlin123]

Sorry for the constant nudge, but just wondering if this PR looks good and can be merged soon?

[alvinlin123]

@markan any chance you would be able to take a look at this PR soon?

[markan]

First of all, sorry for the delayed review.
This looks good; one minor suggestion for clarity.

[markan]

Maybe this instead?

Suggested change

	r, w := 0, 0
	for r < len(msgs) {
	if len(msgs[r]) <= maxMsgLength {
	// Keep it.
	msgs[w] = msgs[r]
	r++
	w++
	continue
	}
	// This message is a large one, so we send it alone.
	bufs = append(bufs, bytes.NewBuffer(msgs[r]))
	r++
	}
	msgs = msgs[:w]
	for r, b := range msgs {
	if len(b) <= maxMsgLength {
	// Keep it.
	msgs[w] = b
	w++
	} else {
	// This message is a large one, so we send it alone.
	bufs = append(bufs, bytes.NewBuffer(b))
	}
	}
	msgs = msgs[:w]

Should be logically equivalent, but removes the need for the continue and captures that we always advance r each step, so it's instantly obvious that the loop completes and looks at each input exactly once.

Or maybe this: (adapted from https://github.com/golang/go/wiki/SliceTricks#filtering-without-allocating)

Suggested change

	r, w := 0, 0
	for r < len(msgs) {
	if len(msgs[r]) <= maxMsgLength {
	// Keep it.
	msgs[w] = msgs[r]
	r++
	w++
	continue
	}
	// This message is a large one, so we send it alone.
	bufs = append(bufs, bytes.NewBuffer(msgs[r]))
	r++
	}
	msgs = msgs[:w]
	w := a[:0]
	for _, b := range msgs {
	if len(b) <= maxMsgLength {
	w = append(w, b)
	} else {
	// This message is a large one, so we send it alone.
	bufs = append(bufs, bytes.NewBuffer(b))
	}
	}
	msgs = w

[Amier3]

Hey @alvinlin123 or @pracucci ,

Are you still able to follow up on these requested changes?

[alvinlin123]

Yes, let me try to see if I can get a hold of @pracucci :-)

[alvinlin123]

No worries @markan, thank you so much for taking a look!

[jeromeinsf]

@pracucci Do you think you could take a look?

[alvinlin123]

@Amier3 @markan I have addressed the PR review comments in separate PR: #262

Please let me now if #262 looks ok.