Transition value JSON from old to new flat format

[suvayu]

Migrate old JSON parameter_values into the new schema that is more like a flat table (for time_series, array, and map) and singular pyarrow compatible values for date_time, duration, and time_pattern.

No related issue

Checklist before merging

• Documentation (also in Toolbox repo) is up-to-date
• Release notes have been updated
• Unit tests have been added/updated accordingly
• Code has been formatted by black & isort
• Unit tests pass

Authors

Since GH doesn't support setting multiple people as author in a PR, documenting it here

@OleMussmann, @suvayu

[suvayu]

Add some notes/questions as review comments.

[soininen]

I think we need to allow null values in ArrayIndex and the like to support uneven maps, i.e. this should be a valid table:

index 1	index 2	value
A	null	1.1
B	null	1.2
C	a	2.1
C	b	2.2

This raises the question whether we need the index arrays as separate types at all.

I also reintroduced custom conversion for some types into models.py to get the unit tests to pass. I am currently working on Toolbox to make it compatible with this branch and for that I need to_database() and from_database() to work.

[suvayu]

Hi,

Strictly speaking, the index column and value column distinction is not needed. But I would like to have them because then downstream code can make useful assumptions. But before I get into that, I think there's a misunderstanding here, mostly because I think I didn't document this anywhere. There is no requirement for all but the last column to be an index column. So this is acceptable:

col1	col2	value
A	null	1.1
B	null	1.2
C	a	2.1
C	b	2.2

It would mean col1 is index type, but col2 and value are just nullable arrays.

The reason this is useful, say, when converting to a dataframe (or something else in a user script), we can treat col1 as index, while excluding col2. Making that assumption won't require inspecting the contents of col2, the type can indicate this is safe.

Of course this is not feasible when working only with the parameter_value, but it is useful when you combine the other columns from the table.

Anyway, I think we should discuss this a bit further. I'll email you.

Cheers,

[soininen]

Strictly speaking, the index column and value column distinction is not needed. But I would like to have them because then downstream code can make useful assumptions. But before I get into that, I think there's a misunderstanding here, mostly because I think I didn't document this anywhere. There is no requirement for all but the last column to be an index column. So this is acceptable:

col1	col2	value
A	null	1.1
B	null	1.2
C	a	2.1
C	b	2.2

It would mean col1 is index type, but col2 and value are just nullable arrays.

These are very good points. I agree that the first index column (col1 in the example) should not be nullable. We should indeed keep the index column types.

[soininen]

Could this class be replaced by typing.NewType?

TimePeriod = NewType("TimePeriod", str)