Investigate non-blocking tracing subscriber to improve TPS performance

[gabriel-aranha-cw]

Description:

Currently, our tracing implementation appears to be causing TPS degradation due to blocking log operations. We should investigate using a non-blocking tracing subscriber to minimize the performance impact of logging.

Current Situation

• Using blocking tracing subscriber
• Observable TPS degradation when logging is enabled
• Currently reaching ~2.5k TPS for writing operations

Investigation Points

• Research available non-blocking tracing subscriber implementations

Success Criteria

• Measure and compare TPS with current vs non-blocking implementation
• Minimize impact on transaction processing performance

References

• tracing crate documentation
• tracing-subscriber documentation

[carneiro-cw]

I tried setting the writer of the stdout layer to be non-blocking here #2051 but this did not seem to make any difference. Still need to investigate if the other layers are interfering with this.