Max batch sizes don't allow for headers

[davidbending]

The maximum size of batches in SendPartitionedBatchAsync for queues and topics sets a maximum size of 256k, but this doesn't allow any space for the header. Because i have a couple of message headers I'm getting:

A request from the client instance has exceeded the maximum message size, and the underlying channel will be recreated. Validate the content size before retrying.

I've fixed up my code by allowing for up to 32k of header but this is something other people might want to be aware of.

[paolosalvatori]

Hi David
sorry for the huge delay. Could you please share your fix? Thanks!
Ciao
Paolo

[SeanFeldman]

This is what I've noticed:

• the only data taken into consideration by BrokeredMessage.Size is body (before send operation)
• size of custom properties is ignored
• size of standard properties (especially those that are string based like SessionId, ReplyTo, MessageId, etc.) is ignored
• full message size is reported by BrokeredMessage.Size after message is sent

I was thinking to follow the approach that @davidbending took, but in my case custom properties can be really big (an exception with the full stacktrace from an async code - easily can get above 32K).

[paolosalvatori]

Thanks Sean
I'll eventually fix the bug, but I'm quite busy in this period. It would be great if you could fix this solution as you suggested above. :)
Ciao
Paolo

[SeanFeldman]

@paolosalvatori I was looking into this and not sure it's that trivial. Here are my findings.

To perform message size estimated calculation, 3 things need to be taken into consideration: body, custom properties, and standard properties. Standard properties are not an issue, they are typed and size is pre-determined or can be calculated (in case of string properties). Custom properties could be tricky, depends what types are stored and being able to determine that type. With the body there's a problem. To get a body of a BrokeredMessage, that message has to be cloned first. Once message is cloned, there's a need to know if message was a Stream or not. Assuming that body type somehow can be figured out, for a body of non-Stream type, to estimate the size the body has to be serialized using the same serialization BrokeredMessage is using. That serializer is internal to ASB. At this point I got stuck.

[paolosalvatori]

Got it. Thanks. Let's see if Dan Rosanova says anything about this on Yammer :)

[SeanFeldman]

The answer was clear - requires new functionality.
I'd think there're 2 things: new name (for existing method) and a new functionality 😄

I have build something that achieves this, estimates message size with a controllable padding percentage, but it requires raw data before its turned into a BrokeredMessage. Once data is in the BrokeredMessage shape, it's a little too late.

[paolosalvatori]

Yes...Indeed, as you said, this feature should be provided OOB by the client library. Any workaround is just an approximation of the real solution, so I don't know how much it's worth it to create a general purpose solution that tries to fit all needs. And I'm too busy to spend time on this. :) It's Dan business to solve the problem. :)

[SeanFeldman]

I'm going to blog on this, showing how I solved this. Hopefully this will help those that need to implement it themselves. Until ASB team does it :)