Is there a way to retrieve type from file?

[alex-s-gardner]

I realize that reading from a netcdf is not not type-stable:
https://juliageo.org/NCDatasets.jl/stable/performance/#performance_tips

When using NCDatasets I typically identify the variable type by asserting an incorrect type and letting typeassert correct me as to what the true type is:

My workflow looks like this:

  using NCDatasets
  using Downloads
  path2nc = "https://people.sc.fsu.edu/~jburkardt/data/netcdf/part.nc"
  path2nc = Downloads.download(path2nc)
  ds = NCDataset(path2nc)
  
  B = ds["Tetrahedron_Vertex"][:,:]:: Array{Float64, 2};

  ERROR: TypeError: in typeassert, expected Matrix{Float64}, got a value of type Matrix{Int32}
  Stacktrace:
   [1] macro expansion
     @ ./timing.jl:689 [inlined]

    B = ds["Tetrahedron_Vertex"][:,:]::Matrix{Int32}

The type assertion error is returned instantly where as loading a large NC file without typeassetion can take a very long time.

Having to manually assert the type is rather cumbersome for more generic code.

This makes me think that there must be a way to quickly retrieve the type from the file before reading, if non is provided by the user. I'm sure this has been explored and I'm just wondering what the reasons are for no implementing.

[Alexander-Barth]

You can use T = eltype(ds["Tetrahedron_Vertex"]) (this does not load the data). You likely still need barrier function to avoid spreading of a type instability:

# bad example
T = eltype(ds["Tetrahedron_Vertex"]) # still unstable, because the julia compiler will not open the NetCDF file during compile time :-)
B = ds["Tetrahedron_Vertex"][:,:]::Matrix{T} 

With a barrier function (untested) you would do:

function stuff(::Type{T},ds) where T
  B = ds["Tetrahedron_Vertex"][:,:]::Matrix{T} 
  # computing with B
end

T = eltype(ds["Tetrahedron_Vertex"]) 
stuff(T,ds) 

I am not sure if there is really an advantage over this simpler barrier function

function stuff2(B)
  # computing with B
end

stuff2(ds["Tetrahedron_Vertex"][:,:])

[visr]

In time it might also be interesting to add a type stable API similar to https://github.com/gbaraldi/StaticHDF5.jl.

[Alexander-Barth]

Note also this related discussion:
#256

But

B = Variable{Int,2}(ds,"Tetrahedron_Vertex")[:,:] # not implemented

is not really shorter than:

B = ds["Tetrahedron_Vertex"][:,:]::Matrix{Int32}

Actually it is the same length :-). But it would help with --trim I guess.

Unfortunately with CFVariable, the user would also need to specify the types of add_offset, scale_factor, time units... (all attributes that potentially changes the type), which would make an unpleasant API.

[alex-s-gardner]

You've clearly thought deeply about this, and reading through the threads i see that there is no easy fix. For pragmatic purposes, what advantages do you get with a type unstable Julia implementation of netcdf vs calling a type stable compiled library? Really for my own edification.

[Alexander-Barth]

The type unstable API allows for more conceice code. If you know the types (and number of dimensions) of the variables that you will read you can also use the type stable in-place NCDatasets.load!.

Feel free to reopen if necessary 😃