Do we really want to allow qualifying array elements INSIDE an input/output occurrence?

[jerstlouis]

@pvretano

There was this change made in Core where valueNoObject.yaml now takes:

  - type: array
    items:
      oneOf:
        - $ref: "inputValue.yaml"
        - $ref: "qualifiedValue.yaml"

instead of (old valueNoObject.yaml):

  - type: array
    items: { }

Why? Would you know which issue number discussed this, if any?

Sorry I was not able to follow all the discussions with everything going on.

The way that input.yaml is structured, it already does:

oneOf: 
  - $ref: "inlineOrRefValue.yaml" 
  - type: array 
    items: 
      $ref: "inlineOrRefValue.yaml" 

So this is what accounts for the multiplicity and qualification of the input values.

So what you would have in inlineOrRefValue.yaml is the input value (individual occurrence) itself.
What this change does now is let you "qualify" things INSIDE the input value (individual occurrence, and in a recursively nested fashion), clashing with the specified JSON Schema in the process description.
This change does not seem right to me.
Before, that generic array was whatever array described in the process description... It's not an input that should be qualifiable or including valueNoObject.yaml again...
That might have something to do with the deep recursion that was breaking the older Swagger UI (I had to migrate to the newer version to get over these errors).

cc. @fmigneault @gfenoy @bpross-52n

[fmigneault]

It forces (not just "let") you to use the qualified input structure if you want to pass an array (cardinality / minOccurs>=1) of complex objects.

Before this change, there is no way to distinguish complex structures that happens to have a value property vs a qualified one, and it makes parsing highly nested structure pure guesses (i.e.: along which nested array/object you can safely scale out the operation).

[jerstlouis]

@fmigneault But the cardinality is handled by input.yaml, is it not?

Say I try to use as input e.g.:

[
   { "foo": 1 },
   { "foo": 2 }
]

That is the minOccurs >= 1...
And inlineOrRefValue forces me to use a qualifiedValue for complex objects.

So that would already be invalid.

I would need to use:

[
   { "value": { "foo": 1 } },
   { "value": { "foo": 2 } }
]

so that "value" at that level would always mean the "qualified object", never an object with "value".

If the object has a "value" property, that is not an issue because the object would always be inside the top-level "value"-qualified object.

(all this is before the change).

Now if your input is maxOccurs = 1 but the type/schema is an array of some object (specified in the items: of the array schema), this would have been valid"

[ { "foo": 1 }, { "foo": 2 } ]

but if maxOccurs >= 1, but the type still an array of object, this would have need to be specified as:

[ [ { "foo": 1 }, { "foo": 2 } ] ]

and even if it's a single object being passed the array is always needed to avoid the ambiguity:

[ [ { "foo": 1 } ] ]

I believe this is what Requirement 19 (from V1) was trying to say:

Conditions
The process input is defined in the process description as having a maximum cardinality of greater than one (maxOccurs>1).
A The server SHALL support process input values encoded as an array.
B This SHALL be true even if the input consists of a single value.

The nested structures, i.e., anything inside the array, or anything inside an object that is an actual output, was entirely left to the process description to specify however it wants. The qualification/multiplicity aspect only applied to the top-level.

Now if the qualifying can happen anywhere, this is extremely confusing, and can easily conflict with the actual schema specified?

[fmigneault]

Your interpretation matches mine (i.e.: you need the array of qualified values and the intent is NOT to nest qualified values). Many of my examples #548 (comment) and probably elsewhere in older issues align with this.

I think one issue was that json-schema validation was failing because there was weird oneOf cases that were matching multiple sub-branches with specific inputs. This was probably an attempt to fix this conflict, but maybe not the "correct" fix.

I'd really like to have a long list of explicit examples where we validate against the schema with the CI each time... to avoid re-introducting such problems.

[jerstlouis]

@fmigneault there was one fix that was applied and good which was to change oneOf to anyOf.

But I think this change does not go in the right direction, and we had spent considerable thinking about and discussing in depth all possible scenarios in the development of v1.0, and I think this was all working correctly, except the language of that Requirement could have been clearer by using language saying:

The server SHALL interpret ... as .... when... etc.

(specifically, saying to interpret as multiple inputs vs. one input vs. array input, based on the maxOccurs and input schema).

and yes this should have several example of all possibilities (and ideally also unit tests / CI for them)

[fmigneault]

Ah! I think I have a lead.

The nested qualified is needed for the multi-results.

ogcapi-processes/openapi/schemas/processes-core/input.yaml

Lines 1 to 5 in 77cec18

	oneOf:
	- $ref: "inlineOrRefValue.yaml"
	- type: array
	items:
	$ref: "inlineOrRefValue.yaml"

ogcapi-processes/openapi/schemas/processes-core/results.yaml

Lines 1 to 2 in 77cec18

	additionalProperties:
	$ref: "inlineOrRefValue.yaml"

Both using:

ogcapi-processes/openapi/schemas/processes-core/inlineOrRefValue.yaml

Lines 1 to 4 in 77cec18

	oneOf:
	- $ref: "valueNoObject.yaml"
	- $ref: "qualifiedValue.yaml"
	- $ref: "link.yaml"

The fix should be that it must have the same oneOf[...inlineOrRefValue...] above allowing single or array variants like inputs do. Then the very-embedded array below does not need to include qualified anymore since it would be handled the same way inputs do.

ogcapi-processes/openapi/schemas/processes-core/valueNoObject.yaml

Lines 7 to 10 in 77cec18

	items:
	oneOf:
	- $ref: "inputValue.yaml"
	- $ref: "qualifiedValue.yaml"

[jerstlouis]

@fmigneault Shouldn't that be output.yaml if we're trying to mirror the inputs?

Or is that output.yaml something completely different? (Yes it's the output SELECTION used in the execution request).
We should rename that file outputSelection.yaml.

A major issue in this directory is that everything is in the same directory, instead of having separte sub-directories for the:

• process description
• execution request
• job management

That would go a LONG way into untangling all this.

If the (JSON) output really has the same multiplicity as the inputs, then we would have the NEW output.yaml that maybe could be inputOrOutput.yaml (rename current input.yaml -> inputOrOutput.yaml), and then results.yaml would be:

additionalProperties:
  $ref: "inputOrOutput.yaml"

and then we could revert those problematic changes to valueNoObject.yaml.

[fmigneault]

output.yaml is for the process description, the "requested outputs" and which format they should use.
The values will be contained in the results.yaml, either in execution sync or job async.

(would help to have explicit description in those schemas, since I often get confused by it as well)

Probably clearer to just have "value" or "data", and avoid the input/output names altogether. It makes it clearer that it is not the I/O descriptions but the actual data contents.

[jerstlouis]

@fmigneault

output.yaml is for the process description, the "requested outputs" and which format they should use.

Yes this is why I suggest we rename this to outputSelection.yaml to avoid very significant confusion.

Probably clearer to just have "value" or "data", and avoid the input/output names altogether.

I like inputOrOutput since we already use value in several places like inlineOrRefValue / valueNoObject / qualifiedValue, so "inlineOrOutput" would clarify that this is the shell with the multiplicity considerations.

(and we probably need to rename inputValue.yaml to anyValue.yaml or something to avoid the confusion since it's also used in outputs now).

[jerstlouis]

@fmigneault

output.yaml is for the process description, the "requested outputs" and which format they should use.

It's for the execution request, it's not used in the process description. That uses outputDescription.yaml.
Hence why I suggest we should split this schemas/ directory into sub-directories to avoid such confusion.

[fmigneault]

inlineOrRefValue / valueNoObject / qualifiedValue

This is precisely why I would just use value.yaml, since those have no mention of inputs/outputs, but "value". Then, inputs.yaml and results.yaml can include value.yaml and it remains very clear.

Yes this is why I suggest we rename this to outputSelection.yaml to avoid very significant confusion.

Yes, that's good as well. Could have description to be even more explicit.

It's for the execution request, it's not used in the process description. That uses outputDescription.yaml.

Correct. My bad. Always confusing 😅

[jerstlouis]

@fmigneault

It is not clear, because for me "value" is the thing "inside" the multiplicity shell, while inputOrOutput is the multiplicity shell for inputs/outputs. We could call it oneOrMultipleValues if you like instead.

(but I still think this could be confusing... it's the "top-level multiplicity shell to receive input values or return output values". not something that can be nested, which are arguably also "values")

(that is input.yaml currently)

and I guess you meant execute.yaml, we don't currently have an inputs.yaml.

[francescoingv]

I would like to ask some aspects on the following practical example with mixed types.

@fmigneault : as I understand in the following object_array is always a single array, possibly with one single element.

  'inputs': {
    'simple_number': {
      'title': 'numberExample',
      'schema': {
        "type": "number"
      },
      'minOccurs': 1,
      'maxOccurs': 1
    },
    'object_array': {
      'title': 'arraysOfObjects',
      'schema': {
        "type": "array",
        "minItems": 1,
        "items": {
          "type": "object",
          "required": ["item1", "item2"],
          "additionalProperties": False,
          "properties": {
            "item1": {
              "type": "number"
            },
            "item2": {
              "type": "number"
            }
          }
        }
      },
      'minOccurs': 1,
      'maxOccurs': 1
    },
    'single_object': {
      'title': 'single object,
      'schema':{
        "type": "object",
        "required": ["item3", "item4"],
        "additionalProperties": False,
        "properties": {
          "item3": {
            "type": "number"
          },
          "item4": {
            "type": "number"
          }
        }
      },
      'minOccurs': 1,
      'maxOccurs': 1
    },
  },

My understanding is that a valid input is:
{ "simple_number": 5 , "object_array": [{ "item1": 1 , "item2": 2 } ] , "single_object": { "value": { "item3": 3 }, { "item4": 4 } } ]
while the following is not valid:
{ "simple_number": 5 , "object_array": [{ "value" : { "item1": 1 , Item2": 2 } } ] , "single_object": { "value": { "item3": 3 }, { "item4": 4 } } ]

Also, I understand an echo process would either return:
{ "simple_number": 5 , "object_array": [{ "item1": 1 , "item2": 2 } ] , "single_object": { "value": { "item3": 3 }, { "item4": 4 } ]
or
{ "simple_number": 5 , "object_array": [{ "item1": 1 , "item2": 2 } ] , "single_object": { "reference": { "$ref": "someURL" } } ]

and the output can be used as input to the same echo process.