Explore https://www.openstoragenetwork.org/ for storage

[yarikoptic]

I was told it provides s3 API so could be storage backend for a dandi instance?

Uses incommon auth so could be a good way for institutional labs I guess,

TODOs to check

• support for tagging and then support for ACL based on tags to be setup "bucket wide" - as we do for embargo access control
• support for versionId's

[CodyCBakerPhD]

OSN (in particular, storage costs) is described in the combined S3 backup doc: https://github.com/CodyCBakerPhD/dandi-archive/blob/all_backup/doc/design/all_backup_options.md#cost-summary

Needless to say that would be cheaper than storing objects on any non-Glacier tier of an S3 bucket, but what is throughput like?

[yarikoptic]

I believe a local PI getting one of those, so we could try but if I got topology right, we would have "good enough" (10Gbps) pipe between us and the warehouse. In general it would be "institution specific". The main point would be to verify that whatever that S3 API is that it is "compatible" with what we use/expect of S3 API as IIRC there could be differences and hence we have e.g. some minio specific adapters:

❯ git grep 'class .*DandiS3Storage'
dandiapi/storage.py:class DandiS3Storage(S3Storage):
dandiapi/storage.py:class MinioDandiS3Storage(DandiS3Storage):

[CodyCBakerPhD]

Guessing the biggest aspect would be application/reliance on tags? I have no idea how S3 treats/stores that info behind the scenes for objects on a bucket

[yarikoptic]

do you mean ETags? yes, that would be one aspect in coordination with upload/chunking iirc.

another -- versioning since I do rely on versionIds for datalad2backup and likely we will rely on for zarr manifests based solution (already relying in current prototype we have via https://webdav.dandiarchive.org/)

then minting upload urls etc, might be of relevance.

[CodyCBakerPhD]

do you mean ETags?

No, I mean object tags. Like versions. And embargo.

[yarikoptic]

I thought that versionId is a magical dedicated thing, did not know it is just an instance of some generic 'tag'...

embargo -- that's a great point, indeed, worth checking if such policies could be set up. Adding into TODOs

[CodyCBakerPhD]

Ahhh maybe 'versioned buckets' is indeed an S3 configuration separate from tags, IDK the exact specifics of that. But you could still use custom tags for that purpose if you don't enable bucket versioning