Try to avoid allocations

[Kixunil]

There's some code that allocates when not required (or forces allocation onto users). While cryptography is the bottleneck it might be better not to push it too much. We may find that some people do something that is slow because of the allocations and in some cases it's not super-hard to just avoid them.