Support multi-part uploads for Zarrs

[yarikoptic]

Prompted by

• Validation error related to content-md5 when uploading OME-Zarr dandi-cli#1821

ATM we use single part "put_object" for zarrs:

92 class ZarrViewSet(ReadOnlyModelViewSet):
283 def create_files(self, request, zarr_id):
286 with transaction.atomic():
301 urls = [
302: zarr_archive.storage.generate_presigned_put_object_url(

The immediate use case of LINC project (attn @kabilar @satra) apparently has multi-GB files within zarrs!

Max single-part upload size is 5GB and there dandi-cli had succeeded with 2.6GB upload and potentially others were even larger (apparently we do not depict error properly there if that was the case).

In any case -- with v3 sharding it is logical to expect large files within zarr "filesets". Hence we need to implement support for multi-part uploads, and likely as consistent with regular assets uploads as possible/feasible.

Components affected and requiring analysis/changes:

• dandi-archive --
• dandi-cli --
• backups2datalad -- likely we use for minting keys and/or checks
• dandidav -- "not sure" -- it might in particular in case of manifests and checksums/ETags there
• Add versioned Zarr design doc #2702 -- same as per dandidav
• ...?

Also worth checking aiming forward:

• what are the file sizes within nwb-zarr's AIND works with (@alejoe91 ?)

[satra]

i didn't realize zarr uploads used single part only. indeed a shard could be terabytes in size, and it's possible that a given zarr object is uploaded as a single shard. i.e. we would allow someone to upload a 5TB shard (the limit of s3).

[kabilar]

i.e. we would allow someone to upload a 5TB shard (the limit of s3).

S3 recently increased the maximum object size to 50TB: https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-s3-maximum-object-size-50-tb/

[alejoe91]

what are the file sizes within nwb-zarr's AIND works with (@alejoe91 ?)

Hi guys! Currently not that large. We are currently not saving raw traces (we will eventually when we generate files to upload to DANDI), but only LFPs/units (downsampled in time and space). So the average NWB file(for ecephys) is about 2-300 MB.

@arielleleon what about ophys?

[arielleleon]

@yarikoptic NWB files for ophys are aroung 1-2GB

[yarikoptic]

thanks @arielleleon . In other words -- in the order of the limit of 5GB, so might reach it.

[satra]

is there any reason why all uploads (of any object) can't be multipart by default, even if it ends up being 1 part?

[yarikoptic]

I do not see why "can't" technically and likely that is where we should go.

We probably (someone could do archaeological expedition) had reasoning for easier matching to ETag of verification of uploads, may be before we made it more robust for regular asset blobs (which now have also async job to compute and potentially verify checksums IIRC, and I do not think we do that for zarrs). @waxlamp @jjnesbitt do you have recollection(s) on that?

[kabilar]

Max single-part upload size is 5GB and there dandi-cli had succeeded with 2.6GB upload and potentially others were even larger (apparently we do not depict error properly there if that was the case).

Hi @yarikoptic, I might be misunderstanding, but we have asset blobs that are larger than 5GB: https://dandiarchive.org/dandiset/001602/draft/files?location=derivatives/micr&page=1 (embargoed dandiset)

[kabilar]

Hi @waxlamp @jjnesbitt, following up here to see what the next steps are to enable multipart upload of Zarrs? Thanks.

[jjnesbitt]

Hi @waxlamp @jjnesbitt, following up here to see what the next steps are to enable multipart upload of Zarrs? Thanks.

We'll need to somewhat replicate the multi-part upload logic for zarrs, since the existing process for Uploads result in AssetBlob objects, while this workflow would not. I'm actively working on it.