wire: separate protocol message limit from serialization bound

[erickcestari]

Revert MaxMessagePayload to 32MiB and introduce MaxProtocolMessageLength (~4MB) for p2p network message size enforcement. This mirrors Bitcoin Core's separation between MAX_SIZE (32MiB serialization bound) and MAX_PROTOCOL_MESSAGE_LENGTH (~4MB network limit) introduced in bitcoin/bitcoin#5843.

The previous commit reduced MaxMessagePayload to 4MB, but that constant is also used as a serialization bound for deriving maxTxInPerMessage, maxTxOutPerMessage, and variable-length string limits in contexts beyond network messages (e.g. database deserialization via MsgTx.Deserialize). While consensus limits keep real values well below the 4MB-derived bounds, conflating the two constants is architecturally incorrect and diverges from Bitcoin Core's design.

The new MaxProtocolMessageLength is now enforced in all four network read/write paths: WriteMessageN, WriteMessageWithEncodingN, ReadMessageWithEncodingN, and ReadV2MessageN (which previously had no overall message size check).

closes #2503

[Roasbeef]

LGTM 🥦

[kcalvinalvin]

Shouldn't we also replace with MaxMessagePayload here:

btcd/wire/msgreject.go

Lines 172 to 180 in b950e5b

	// The reject message did not exist before protocol version
	// RejectVersion.
	if pver >= RejectVersion {
	// Unfortunately the bitcoin protocol does not enforce a sane
	// limit on the length of the reason, so the max payload is the
	// overall maximum message payload.
	plen = MaxMessagePayload
	}

[kcalvinalvin]

nit: Everywhere else we return messageError() but here we return fmt.Errof(). Might be worth aligning to avoid error type check errors later down the road

[erickcestari]

ReadV2MessageN function it's only using fmt.Errorf(), but I'll refactor to return messageError() instead.

[erickcestari]

Shouldn't we also replace with MaxMessagePayload here:

Sure, but I thinks is work for a follow-up PR to audit all MaxPayloadLength implementations and ensure none exceed MaxProtocolMessageLength. The MsgReject case isn't a bug since the protocol-level check in the read/write paths already caps messages at 4MB, but aligning per-message limits is worth its own review since it touches many files.