Dear all,
currently I am diving into the use of spatialdata and are now exactly at the point where I wanted to use compression in storage options. I am not sure if this fits here in the issue. If not let me know and I will ask somewhere else.
Here is my scenario:
I have standard visium data, not a problem which I want to align with several wsi stored in the vsi format. This format uses compression (i guess jpeg) to keep the data small. This is debatable but makes the data still handy and well enough readable for our research. Now my workflow i the following. I export the data deposited on an omero instance via omero zarr export and read them except some metadata details with :

import os
from ome_zarr.io import parse_url
from ome_zarr.reader import Reader
import dask.array as da
import xarray as xr
from spatialdata.models import Image2DModel
from spatialdata.transformations import Scale
from spatial_image import SpatialImage

# exists from visium import
spatial_data

...
reader = Reader(parse_url(zarr_path))
....


# read in the highest resolution
raw = da.from_zarr(f"{zarr_path}/0")
img_array = _slice_to_2d(raw)
xr_img = xr.DataArray(img_array, dims=("c", "y", "x"))
sp_img = SpatialImage(xr_img)

# use Image2DModel parse to load the data and generate different pyramidal resolutions with spatial_data 
img_model = Image2DModel.parse(
   sp_img,
   transformations={marker: Scale([x_scale, y_scale], ["x", "y"])},
   rgb=None,
   chunks=(img_array.shape[0], 256, 256),
   scale_factors=[2] * number_of_scales
)
spatial_data.images[marker] = img_model

Now after adding several images with the name marker I use
spatial_data.write()

to generate a portable self-contained file that has all information. But this now has two main problems. First the number of inodes on a HPC explode (i would love to write to .zip stores) but also the size is now >100 Gb per Visium experiment. Even though everything is there we still cannot really work with these datasets. Maybe we just need bigger computers.
What is the recommended workflow?

My first idea was to use spatial_data.write(compression=jpeg2000) or something. Or should I write down the wsi somewhere else and not store everything with the high level .write() function? But if so I would be grateful (to also participate in generating) about a documention/tutorial to help people organize their data appropriately.

Re-enable compression #37

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions