Add support for validating Zarr folders with format V3 using custom walk-through and tensorstore by candleindark · Pull Request #1614 · dandi/dandi-cli

candleindark · 2025-04-16T17:22:17Z

This PR brings the following changes

It provides a temporary workaround to support validation of Zarr format V3 objects, Zarr arrays or groups, before zarr-python 3.x upgrade is done. It closes zarr 3 support #1609.
In achieving point 1, this PR provides a new definition of get_zarr_format_version() provided and uses this definition. It closes Make use of get_zarr_format_version to provide version of zarr #1610.
Additionally, this PR corrects the use of zarr.open() in validation. zarr.open() has a default mode of a, read and write. In this mode, zarr.open() would attempt to modify a Zarr store if it deems the store as an invalid one, corrupting the store. This PR changes the use of zarr.open() from a mode to r mode, read-only.

Note:
Zarr format 3 objects/stores are added to the git repo for testing since with zarr-python 2.x, we can't generate such stores dynamically.

Remaining TODOs:

Add/update tests

codecov · 2025-04-16T17:24:44Z

Codecov Report

Attention: Patch coverage is 85.13514% with 11 lines in your changes missing coverage. Please review.

Project coverage is 88.69%. Comparing base (88c3275) to head (8ba6a8f).
Report is 96 commits behind head on master.

Files with missing lines	Patch %	Lines
dandi/files/zarr.py	82.25%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1614      +/-   ##
==========================================
+ Coverage   88.58%   88.69%   +0.11%     
==========================================
  Files          78       78              
  Lines       10937    10991      +54     
==========================================
+ Hits         9688     9749      +61     
+ Misses       1249     1242       -7

Flag	Coverage Δ
unittests	`88.69% <85.13%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dandi/files/zarr.py

So that it can handle both Zarr format V2 and V3

A top level func to validate a Zarr format V3 LocalStore

This is no longer needed. Individual arrays are located within `_ts_validate_zarr3()`

dandi/files/zarr.py

A function to validate a Zarr format V3 array in a LocalStore with the tensorstore

To avoid import of dandi when there is already `from dandi import ...`

Zarr format V3 handling is done though tensorstore

It is no longer needed for all Zarr validation errors are now return as a `ValidationResult`

This is to achieve consistency with other validation funcs in the package

candleindark · 2025-04-17T19:25:12Z

@yarikoptic Is a Zarr store, the *.zarr directory in the filesystem the represent a Zarr object, array or group, considered as a file in dandi-cli? I see code such as https://github.com/dandi/dandi-cli/blob/6518334b60c541890d83e293d000f350280dfbad/dandi/files/zarr.py#L265C21-L265C37 treat a directory as file by using Scope.FILE to mark a validation result.

Currently, I used Scope.FOLDER to mark validation error regarding Zarr objects. Should I be consistent and use Scope.FILE intead?

yarikoptic · 2025-04-17T19:30:13Z

File scope was used to reflect the fact that it applies to that single "Zarr dataset", for which we have a single asset in DANDI, and not a directory which might contain multiple files/assets... Let's go with File for now, although we might want to improve or change semantic here later on

So to be consistent with other usage of the mark in this project

`zarr.open()` have a default mode of `a`, read and write. In this mode, `zarr.open()` would attempt to modify a Zarr store if it deems the store as an invalid one, corrupting the store

So that Pydantic doesn't apply any type coercion in validation

candleindark · 2025-04-17T23:00:19Z

@yarikoptic This PR is ready for review. The remaining lines to be tested are either too simple or too tricky to worth investing time for testing this temporary workaround.

github-actions · 2025-04-23T18:55:07Z

🚀 PR was released in 0.68.0 🚀

yarikoptic reviewed Apr 16, 2025

View reviewed changes

dandi/files/zarr.py Outdated Show resolved Hide resolved

yarikoptic reviewed Apr 16, 2025

View reviewed changes

dandi/files/zarr.py Outdated Show resolved Hide resolved

yarikoptic changed the title ~~Add support of Zarr format V3~~ Add support for validating Zarr format V3 using custom walk-through and tensorstore Apr 16, 2025

yarikoptic changed the title ~~Add support for validating Zarr format V3 using custom walk-through and tensorstore~~ Add support for validating Zarr folders with format V3 using custom walk-through and tensorstore Apr 16, 2025

candleindark force-pushed the zarr-format-v3 branch from 010ac61 to 3e1f366 Compare April 17, 2025 01:21

candleindark added 2 commits April 16, 2025 18:41

fix: redefine get_zarr_format_version()

4a98bda

So that it can handle both Zarr format V2 and V3

feat: define func to get zarr arrays in zarr 3 objects

823c13c

candleindark force-pushed the zarr-format-v3 branch from 3e1f366 to 823c13c Compare April 17, 2025 02:15

candleindark added 3 commits April 16, 2025 22:16

feat: define func to get zarr arrays in zarr 3 objects

413b805

feat: implement _ts_validate_zarr3()

ed91298

A top level func to validate a Zarr format V3 LocalStore

feat: remove _get_arrays_zarr3()

c8b84c8

This is no longer needed. Individual arrays are located within `_ts_validate_zarr3()`

github-advanced-security bot found potential problems Apr 17, 2025

View reviewed changes

dandi/files/zarr.py Fixed Show fixed Hide fixed

candleindark force-pushed the zarr-format-v3 branch 2 times, most recently from e3a0735 to e4233da Compare April 17, 2025 17:14

candleindark added 6 commits April 17, 2025 11:50

feat: define _ts_validate_zarr3_array()

99b6e87

A function to validate a Zarr format V3 array in a LocalStore with the tensorstore

rf: eliminate the use of dandi.__version__

e456db3

To avoid import of dandi when there is already `from dandi import ...`

feat: add Zarr format V3 handling in ZarrAsset.get_validation_errors

d34da69

Zarr format V3 handling is done though tensorstore

feat: remove ZarrValidationError

3dd20c6

It is no longer needed for all Zarr validation errors are now return as a `ValidationResult`

feat: add devel_debug param to _ts_validate_zarr3()

8808498

This is to achieve consistency with other validation funcs in the package

feat: add devel_debug param to _ts_validate_zarr3_array()

98d7a86

This is to achieve consistency with other validation funcs in the package

candleindark force-pushed the zarr-format-v3 branch from e4233da to 98d7a86 Compare April 17, 2025 18:50

candleindark added 2 commits April 17, 2025 12:40

fix: use Scope.FILE to mark validation result about Zarr assets

4f1c04b

So to be consistent with other usage of the mark in this project

fix: correct use of zarr.open() in validation

38b4e6d

`zarr.open()` have a default mode of `a`, read and write. In this mode, `zarr.open()` would attempt to modify a Zarr store if it deems the store as an invalid one, corrupting the store

candleindark added bug Something isn't working zarr enhancement New feature or request labels Apr 17, 2025

yarikoptic added the minor Increment the minor version when merged label Apr 17, 2025

candleindark added 2 commits April 17, 2025 15:32

test: provide tests for validation of Zarr format 3 store

0754441

fix: Make _Zarr3Metadata a strict model

8ba6a8f

So that Pydantic doesn't apply any type coercion in validation

candleindark force-pushed the zarr-format-v3 branch from 5562047 to 8ba6a8f Compare April 17, 2025 22:32

candleindark marked this pull request as ready for review April 17, 2025 23:01

candleindark requested a review from yarikoptic April 17, 2025 23:01

yarikoptic added the release Create a release when this pr is merged label Apr 23, 2025

yarikoptic merged commit 1524188 into dandi:master Apr 23, 2025
43 of 48 checks passed

yarikoptic mentioned this pull request Apr 23, 2025

zarr 3 support #1609

Closed

github-actions bot added the released label Apr 23, 2025

candleindark deleted the zarr-format-v3 branch May 22, 2025 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for validating Zarr folders with format V3 using custom walk-through and tensorstore#1614

Add support for validating Zarr folders with format V3 using custom walk-through and tensorstore#1614
yarikoptic merged 15 commits intodandi:masterfrom
candleindark:zarr-format-v3

candleindark commented Apr 16, 2025 •

edited

Loading

Uh oh!

codecov bot commented Apr 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

candleindark commented Apr 17, 2025

Uh oh!

yarikoptic commented Apr 17, 2025

Uh oh!

candleindark commented Apr 17, 2025

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

candleindark commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

candleindark commented Apr 17, 2025

Uh oh!

yarikoptic commented Apr 17, 2025

Uh oh!

candleindark commented Apr 17, 2025

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

candleindark commented Apr 16, 2025 •

edited

Loading

codecov bot commented Apr 16, 2025 •

edited

Loading