"Overlap-only" mode for merge_cubes

[soxofaan]

merge_cubes currently works in "union" mode: the result covers the union of the label ranges for each dimension, which is useful for use cases where you want to stitch together several data subsets containing same kind of data. The main purpose of the overlap resolver is to have a clean seam between them. You have to be careful what you do in the overlap resolver because it might not be applied to all parts of the result (depending on the degree of overlap). If you go beyond taking the mean or first non-null value, the results might be hard to reason about.

Maybe also useful is to provide a "intersection" or "overlap-only" mode of merge_cubes (through an additional parameter, or as a dedicated process, I'm not sure yet) where you only return the dimension ranges where both cubes exist. It's then easier to reason about the overlap resolver because it's guaranteed to be applied everywhere. It would make merge_cubes also useful for use cases where you fuse different kinds of data.

[m-mohr]

merge_cubes is already pretty difficult to implement and thus I'd argue that a new process would be the better choice. Happy to accept PRs.

[m-mohr]

Today I had the idea that you could set all non-overlapping values to null using the overlap-resolver and retain only the values that actually have two valid values, but then I realized that the overlap resolver is likely only called for overlapping areas AND you could not figure out whether a pixel is simply no-data or outside of the overlapping area. So that doesn't work with the merge_cubes process.

The longer I'm thinking about it the less sure I'm about the new process vs. a new parameter thing. On the one hand, merge_cubes is already so complex, but on the other hand, a new process would be mostly a copy of merge_cubes with the restriction on the overlap and I don't like to copy so much text that then needs to be maintained twice...

What priority is this? Is this required for any (upcoming) use-case (in Platform)?

[soxofaan]

a new process would be mostly a copy of merge_cubes with the restriction on the overlap and I don't like to copy so much text that then needs to be maintained twice...

I think that this new "data fusion" process will be a lot simpler description-wise, because you only have to talk about what happens to the intersection. Everything outside the intersection will be discarded by design.

What priority is this? Is this required for any (upcoming) use-case (in Platform)?

It's low prio for now. I was just playing with the idea, triggered by a user (Open-EO/openeo-python-client#239) who tried to do ndvi calculation differently/wrongly with merge_cubes instead of reduce_dimension.

[soxofaan]

Another use case: https://discuss.eodc.eu/t/blocks-after-division/425
user makes division between two data cubes

s2_cube_swf = s2_cube_water_sum / s2_count

The python client translates this to a merge_cubes with an overlap resolver (x,y) -> x / y.
Where there is overlap, the result is x/y as expected, but where there is no overlap the result one of:

• x where there is only first cube data
• y where there is only second cube data

Obviously this is a very confusing result, especially because the user is not really aware that there is a merge_cubes process going on. Without an "overlap-only" merge_cubes variant the user (or client lib) is forced to do some additional masking, which is tedious and comes with a performance cost.

[m-mohr]

@soxofaan I'm curious: What would happen to the x/y example once we would have two processes? Would it be changed to always use the new process (or parameter)? Otherwise, without the user explixitly stating which behavior to use, it is still unclear what happens, right? And you can't really parametrize a basic math operation... ;-)

By the way: This is something where additional logging would be useful.
-> if overlap function is specified but there's no overlap: log a warning?
-> if overlap is specified but one of the data cube has more data than the other: log an info message that some values are not handled by the overlap function?
Would increase the chance that a user at least can figure out what's going on, right? Maybe even something for the process description.

[soxofaan]

In case of (client-side) translating math expressions like x / y, I would always, unconditionally use the "overlap-only" variant of merge_cubes. The things that the original merge_cubes would add is only confusing, if not wrong.
As mentioned above, the user is not really aware that an x / y expression is translated to a merge_cubes process. The user just expects the result of x / y, so no need to add parts where it evaluates to x or y, I can't even imagine the user would want to opt-in to that.

Client-side-wise, I see the original merge_cubes as a data fusion tool, where the user is explicitly aware of the merge_cubes process and its implications (e.g. having to handle an overlap with an overlap resolver).

By the way: This is something where additional logging would be useful.

are you talking about client-side logging or batch job logs?

if overlap function is specified but there's no overlap: log a warning?

I would not consider this an error situation, so a warning seems a bit to strong. I can imagine data fusion workflows were it's equally likely to have overlap or not.

if overlap is specified but one of the data cube has more data than the other: log an info message that some values are not handled by the overlap function

Isn't that the whole point of merge_cubes: that you know in advance to have parts with overlap and parts without overlap? In that sense, I'd just recommend to always log some kind of stats about the overlap and non-overlap zones.

[m-mohr]

Dev Telco: Define a new process in a PR and see how it goes.

[soxofaan]

Define a new process in a PR

cc @VictorVerhaert let's pick up this task (or at least put it on our planning)

[VictorVerhaert]

Made a first attempt at a new process called combine_cubes here: #540

Happy to receive feedback!