zarr 3 support

[calvinchai]

When validating/uploading a zarr-3 archive. The command failed right away. dandi now uses zarr 2 library, which cannot read zarr 3

(dandi) megaera[0]:001412$ dandi validate  --min-severity HINT
[zarr.empty_group] /home/scratch/001412/test.zarr — Zarr group is empty.
2025-04-10 15:09:25,914 [    INFO] Logs saved in /homes/5/kc1708/.local/state/dandi-cli/log/2025.04.10-19.09.24Z-3985259.log
ERROR conda.cli.main_run:execute(125): `conda run dandi validate --min-severity HINT` failed. (See above for error)

[kabilar]

Thanks @calvinchai. For now on the LINC project, please use the --validation ignore flag.

[kabilar]

Migration guide to v3: https://zarr.readthedocs.io/en/stable/user-guide/v3_migration.html

[yarikoptic]

ATM it is (partially?) blocked also by

• [Feature]: Support zarr-python v3 hdmf-dev/hdmf-zarr#202

In principle we could potentially allow installation with Zarr v3 for cases without nwb in those dandisets..

[satra]

@yarikoptic - one possibility in this case is to use tensorstore in the cli (which can read/write both v2 and v3 - and could be more efficient), and thus decouple it (for now) from hdmf-zarr.

[yarikoptic]

Looking at this particular case -- I think we would be just fine with whatever support zarr library 2. provides for opening zarr v3. @candleindark let's address this by

• figuring out what would be an alternative to if isinstance(data, zarr.Group) and not data validation test for having some data for v3
• actually start using get_zarr_format_version function (apparently unused)
  • need to make sure that it works correctly on v3 zarrs! From my quick try it still returned v2 on a sample
  • it is trivial to produce v3 zarr even with 2.11.x library -- see https://zarr.readthedocs.io/en/v2.18.1/api/v3.html
• make sure code works with both v2 and v3 zarr (add / improve tests to create/operate on v3 zarr) and both versions of zarr library

[candleindark]

A few notes here:

1. To enable the Zarr V3 Specification in the zarr Python package at V2. The env var, ZARR_V3_EXPERIMENTAL_API must be set to 1.

Details

➜  scratches pyenv shell dandi-cli
(dandi-cli) ➜  scratches python scratch_2.py
Traceback (most recent call last):
  File "/Users/isaac/Library/Application Support/JetBrains/PyCharm2024.3/scratches/scratch_2.py", line 6, in <module>
    z1 = zarr.open(
         ^^^^^^^^^^
  File "/Users/isaac/.pyenv/versions/dandi-cli/lib/python3.11/site-packages/zarr/convenience.py", line 123, in open
    return open_array(_store, mode=mode, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/.pyenv/versions/dandi-cli/lib/python3.11/site-packages/zarr/creation.py", line 685, in open_array
    z = Array(
        ^^^^^^
  File "/Users/isaac/.pyenv/versions/dandi-cli/lib/python3.11/site-packages/zarr/core.py", line 134, in __init__
    assert_zarr_v3_api_available()
  File "/Users/isaac/.pyenv/versions/dandi-cli/lib/python3.11/site-packages/zarr/_storage/store.py", line 48, in assert_zarr_v3_api_available
    raise NotImplementedError(
NotImplementedError: # V3 reading and writing is experimental! To enable support, set:
ZARR_V3_EXPERIMENTAL_API=1

2. After the ZARR_V3_EXPERIMENTAL_API is set to 1, the zarr Python package will still raise a warning.

Details

```shell (dandi-cli) ➜ scratches python scratch_2.py /Users/isaac/.pyenv/versions/dandi-cli/lib/python3.11/site-packages/zarr/_storage/store.py:39: FutureWarning: The experimental Zarr V3 implementation in this version of Zarr-Python is not in alignment with the final V3 specification. This version will be removed in Zarr-Python 3 in favor of a spec compliant version. warnings.warn( ```

3. The previous two points are related to the following warning in https://zarr.readthedocs.io/en/v2.18.1/api/v3.html.

The experimental v3 implementation included in Zarr Python >2.12,<3 is not aligned with the final V3 specification. This version is deprecated and will be removed in Zarr Python 3.0 in favor of a spec compliant version.

4. get_zarr_format_version() needs to be improved. It is not able to handle the Zarr V3 Specification as implemented by the Zarr Python v2 package.

[yarikoptic]

Yep, I saw the info about env var. If we verify that it provides sufficient support for our minimal tasks, we can set it before importing zarr but after checking its version using packaging module or alike

[candleindark]

I found a few issues with using the zarr Python package at V2 for the Zarr V3 Specification. One among them, in my opinion, disallows us from using it to read Zarr objects/files that are of the Zarr V3 Specification. It is the fact that the zarr Python package at V2 is actually incompatible with the final Zarr V3 specification.

Consider the following files and commands.

create.py

# To be used for creating a Zarr object with Zarr V3 Specification using the
# zarr Python package at version 3 or later

import zarr
import numpy as np

z1 = zarr.create_array(
    store="data/example-1.zarr",
    shape=(10000, 10000),
    chunks=(1000, 1000),
    dtype="int32",
)


z1[:] = 42
z1[0, :] = np.arange(10000)
z1[:, 0] = np.arange(10000)

z2 = zarr.open("data/example-1.zarr", mode="r")
print(z2.info)


# Can be invoked with the following command:
# python3.11 -m venv .venv && source .venv/bin/activate && pip install zarr~=3.0 && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 create.py; deactivate; rm -rf .venv

read.py

# To be used for reading a Zarr object with Zarr V3 Specification using the
# zarr Python package at before version 3

import zarr

z1 = zarr.open("data/example-1.zarr", mode="r", zarr_version=3)

# Can be invoked with the following command:
# python3.11 -m venv .venv && source .venv/bin/activate && pip install "zarr<3.0" && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 read.py; deactivate; rm -rf .venv

Run the following to write a Zarr object in the final Zarr V3 Specification to the file system using the zarr Python package at V3.

python3.11 -m venv .venv && source .venv/bin/activate && pip install zarr~=3.0 && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 create.py; deactivate; rm -rf .venv

Run the following to read the written Zarr object using the zarr Python package at V2, and an error will result due to incompatiblity.

python3.11 -m venv .venv && source .venv/bin/activate && pip install "zarr<3.0" && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 read.py; deactivate; rm -rf .venv

Please note that I used python3.11 because the zarr Python package at V3 requires Python at 3.11 or above.

This result is in agreement with the observation that zarr objects written by the zarr Python package at V2 with the experimental Zarr V3 Specification enabled look very different than those written by the zarr Python package at V3.

[yarikoptic]

We should just generate minimal Zarr v3 with v3 library and commit it to reuse in tests. What error did you encounter, can't run code ATM?

[candleindark]

The error resulted from running

python3.11 -m venv .venv && source .venv/bin/activate && pip install "zarr<3.0" && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 read.py; deactivate; rm -rf .venv

is

Traceback (most recent call last):
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/read.py", line 6, in <module>
    z1 = zarr.open("data/example-1.zarr", mode="r", zarr_version=3)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/convenience.py", line 134, in open
    if contains_array(_store, path):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/storage.py", line 120, in contains_array
    key = _prefix_to_array_key(store, prefix)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/_storage/store.py", line 683, in _prefix_to_array_key
    sfx = _get_metadata_suffix(store)  # type: ignore
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/_storage/store.py", line 592, in _get_metadata_suffix
    return _get_hierarchy_metadata(store)["metadata_key_suffix"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/_storage/store.py", line 587, in _get_hierarchy_metadata
    return store._metadata_class.decode_hierarchy_metadata(store["zarr.json"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/meta.py", line 393, in decode_hierarchy_metadata
    raise ValueError(f"Unexpected keys in metadata. meta={meta}")
ValueError: Unexpected keys in metadata. meta={'shape': [10000, 10000], 'data_type': 'int32', 'chunk_grid': {'name': 'regular', 'configuration': {'chunk_shape': [1000, 1000]}}, 'chunk_key_encoding': {'name': 'default', 'configuration': {'separator': '/'}}, 'fill_value': 0, 'codecs': [{'name': 'bytes', 'configuration': {'endian': 'little'}}, {'name': 'zstd', 'configuration': {'level': 0, 'checksum': False}}], 'attributes': {}, 'zarr_format': 3, 'node_type': 'array', 'storage_transformers': []}

I think it is due to the difference in expectation of data in zarr.json

[yarikoptic]

@candleindark please file an issue against zarr with hope to address that in the 2.x series of the library -- they seems to still release bugfixes there (which is good and responsible approach). Upon a quick search dealing with opening zarr v3 in v2 -- would be good to know if that is anyhow supported or not. If not supported -- there should be a NotImplementedError or alike to inform user.

[candleindark]

@yarikoptic Filed zarr-developers/zarr-python#2981 regarding support of Zarr V3 Specification in zarr-python 2.x and zarr-developers/zarr-python#2982 related to experimental implementation of Zarr V3 Specification in zarr-python 2.x.