Allow configuring max_pool_connections on boto3 s3 client

[ouj]

🚀 Feature Request

Uploading to s3 is running out of pool connections, and the uplink is underutilized

Motivation

WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: <....>.s3.amazonaws.com. Connection pool size: 10

This error keeps happing during my data processing job.

[Optional] Implementation

The key is to add max_pool_connectino config in

streaming/streaming/base/storage/upload.py

Line 247 in 223be8f

config = Config()

.

We can have a BOTO_MAX_POOL_CONNECTIONS environment variable or pass that in from the MDSWriter

Additional context