hydrus can't show nested collections

[vddesai1871]

I'm submitting a

• bug report.
• feature request.

Current Behaviour:

hydrus can't show nested collections. (eg. collection of comments on a particular issue/article)

Expected Behaviour:

Quoting @Mec-iS

we possibly want to return a hyperlink to the collection, not the full collection. same we did for nested object in Flock-Demo.

Possible approach

we can have a hyperlink to the endpoint of nested collection
something like

{ 
"@type": "Article", 
"@id": "api/articles/id", 
... 
"comments": "api/articles/id/comments", 
... 
} 

Steps to reproduce:

Run hydrus with real-world-app API Doc I submitted at HTTP-APIs/hydrus-real-world-app#2 (The API Doc is not complete but can be used to reproduce this issue by adding an article and trying to get/retrieve comments related to that article)

Do you want to work on this issue?

Yes

[chrizandr]

This is true, we need to modify the models as well to be able to this. Properties usually point to instances or terminals. Now we need to consider that they will point to collections. We could create a new relationship model GraphIACol or something that links Instance>Abstract Property>Collection. Inherently we would then have to define a separate model for collections maybe, right now it's just a field in the RDFClass model. Please explore this and work out the details of implementation. We may need to refactor all the models as well to make it less redundant.

[Mec-iS]

This is true, we need to modify the models as well to be able to this. Properties usually point to instances or terminals. Now we need to consider that they will point to collections. We could create a new relationship model GraphIACol or something that links Instance>Abstract Property>Collection.

I don't think this is necessary. It shouldn't be so hard to generate a IRI that points to the collection. The server can read the ApiDoc and find the right collection IRI to point to. Or am I missing something?
It is not the model that needs to represent a property pointing to the collection, but the API that should be able to refer to itself. The server can read the ApiDoc dictionary, find the right resource, add the right id and return the string.

[vddesai1871]

I don't think this is necessary. It shouldn't be so hard to generate a IRI that points to the collection. The server can read the ApiDoc and find the right collection IRI to point to. Or am I missing something?
It is not the model that needs to represent a property pointing to the collection, but the API that should be able to refer to itself. The server can read the ApiDoc dictionary, find the right resource, add the right id and return the string.

Here we are trying to find a way to get a nested collection, this nested collection might be just a small part of the complete collection. For example, the CommentCollection might have 1000s of entries but when a client dereferences api/articles/x/comments, we want to show only those comments which were made on the article with id x, so the server needs a way to identify all these comments(which I think it can't do by just referencing the API Doc). A simple way to do this would be to add an extra field in Instance called parent_id, (In the case of nested collection, when adding a new instance we'll have the parent ID in the URL, here it is x, so parent_id will be set to x), then (on GET) we can directly use it to generate the collection body by searching for all comments with parent_id = x. But this might not be the best way to go around and as we have time I would like to explore other options and the one suggested by Chris.

[Mec-iS]

@vddesai1871
What you describe requires the implementation of the IRITemplate spec feature, as we are supposed to provide in the response a IRI that filters the collection with the expected objects. Again, there is no change to be made in the model, see #302 that should have priority for the first weeks of GSOC. In general, everything that is already in the spec and it is not in hydrus should have priority.

[vddesai1871]

What you describe requires the implementation of the IRITemplate spec feature,

Yeah, I can't believe I completely missed this point-of-view. We'll still need some way to relate elements to their parent.
And intially I was proposing to create a new endpoint/resource for nested collections of form "api/Class/id/NestedCollection" but now by utilizing IRITemplate we can provide a direct IRI api/CommentCollection?parent_id=x, so we'll be able to avoid making major changes in models and there won't be any need to add an extra resource in resources.py. Am I getting it right or missing something?

In general, everything that is already in the spec and it is not in hydrus should have priority.

Couldn't agree more, that way we can avoid over-engineering, and utilize these Spec based features directly for adding any extra functionality.

[Mec-iS]

Am I getting it right or missing something?

Exactly. Please prioritise this #386 and #302 for your first weeks
Also for @Guttz, we need these implemented before starting anything with the demo or the client.

Couldn't agree more, that way we can avoid over-engineering, and utilize these Spec based features directly for adding any extra functionality.

Please take some time to map and list everything we are missing from the spec.

[vddesai1871]

I think the IriTemplate approach we agreed to before can't be used as this issue is not just about showing nested collections but providing complete support for them at hydrus. If I am correct, right now hydrus doesn't provide any support for nested collections, by that I mean that hydrus doesn't provide a way to do any sort of operations over nested collections. The IriTemplate method just provides a way to get nested collection but we also need to provide a way to populate those nested collections. So first of all, hydrus needs to provide a way to add new items to nested collections, it's different than adding items to main collections as it involves getting and setting the parent_id field. We can't expect the client to pass parent_id in the request body, so the only state-less way to do this is having parent_id in the uri.
So the mechanism I mentioned above can be used for that.

For example, the CommentCollection might have 1000s of entries but when a client dereferences api/articles/x/comments, we want to show only those comments which were made on the article with id x, so the server needs a way to identify all these comments(which I think it can't do by just referencing the API Doc). A simple way to do this would be to add an extra field in Instance called parent_id, (In the case of nested collection, when adding a new instance we'll have the parent ID in the URL, here it is x, so parent_id will be set to x)

By setting URIs of members of the nested collection to mainCollection/id this approach will work even if members of nested collection themselves have nested collections as properties.

[Mec-iS]

the server needs a way to identify all these comments

i don't understand why a query on the db cannot return all the comments with article_id X, SQLAlchemy can implent the relationship function to have bi-directional queries. Having explicitly set the parent may help for a graph representation but it should be already possible to query all the comments by article.
The Article object should look like:

...
"comments": ".../articles/3/comments"
...

and the IRItemplate that it links to should run a query that filters by id 3. if article_id is not present in the Comments model, there should be a definition for a backref or a property that allows back referencing to the parent model.

[vddesai1871]

and the IRItemplate that it links to should run a query that filters by id 3.

Yeah, I had agreed to that earlier, but the question is how hydrus add new comments to articles? Don't we need ../articles/3/comments endpoint from where the client can add new comments? If we are having an endpoint for nested collections then there is no need of using IriTemplates.

if article_id is not present in the Comments model, there should be a definition for a backref or a property that allows back referencing to the parent model.

We do not have different models for different classes, we have only one model Instance, so I do not think backref can be used.

Even if Comment class in ApiDoc has an article_id field which might not be true in all cases, hydrus can't infer that article_id is id of the parent element. Hence we need extra generic internal field parent_id.

I believe that with the approach I suggested, we can solve all the corner cases and finally have some mechanism to support any kind of nested collections(with all the possible operations defined in the ApiDoc).

[Mec-iS]

endpoint from where the client can add new comments?

This should happen in the Comment resource, what we have now sounds a bit of un-RESTy. It has been until now for convenience but maybe we can think something better?

We do not have different models for different classes, we have only one model Instance, so I do not think backref can be used.

maybe we should consider having resources classes that extend Instance and not use it directly.

class ArticleInstance(Instance):
   ...

having a generic parent_id sounds good for Instance then.

[vddesai1871]

UPDATE:

This issue is actually about providing support for properties of type hydra:Link(subclass of rdf:Property).
Such Link properties might lead to a collection of resources or an individual resource, it might also involve a list of supportedOperations which might be somewhat different than supportedOperations of the resource it links to.(for example Comment might have issue property which links to the parent issue of that comment and Issue resource in ApiDoc might define all possible operations as supportedOperations but for obvious reasons the Comment/issue (Link) property might only have GET and POST operations as supoortedOperations).
As mentioned earlier this properties can link to a collection or a singal resource, and the only way by which hydrus can find out( or construct) the value (uri) of these properties is by looking at the ApiDoc and finding the range of the particular property. If range is some particular resource(eg. Issue) then hydrus can simply set the value to uri of that resource(which was provided while creating the Comment instnace). If range is some collection then value would be an IriTemplate filtering the collection.
Link to the vocab which I used for examples, http://www.markus-lanthaler.com/hydra/api-demo/vocab

[xadahiya]

This should happen in the Comment resource, what we have now sounds a bit of un-RESTy. It has been until now for convenience but maybe we can think something better?

I agree with @Mec-iS here. Comments should be added to the CommentCollection endpoint only. Doing it any other way is not restful.

Using the Issues and comments vocab example here, Everything will work as it does right now i.e

• Issues will be added on the IssueCollection endpoint
• Comments will be added on CommentCollection endpoint
• Users will be added on the UserCollection endpoint

The only changes that I think we need to implement are -

• Each comment will have a parent associated with it. This can easily be defined in the APIDoc and regarding the client, in any case, the client must know the article details it is adding the comment to, so it should be able to pass the parent ID while creating the comment object to be added. (I can't think of a scenario where someone will try to add a comment without knowing which article/issue it should be added. You can think of something please do let me know.)
• The IRI referencing the comments will use the article ID for example /Issues/3/comments, then hydrus will use the id 3 to generate a partialCollection to retrieve the comments with parent id 3. (The IRI generation, in this case, is really simple. It'll just be <Collectiontype>/<ID>/<NestedCollectiontype>)

My only concern is efficiency here. Filtering and fetching all objects from a collection with a specific property value will be quite inefficient especially in the case where the collection is very large (We'll have to go through the entire collection).
There is one other solution where instead of storing the parent id in each comment object, we can store the comment ids in the parent (can be done by just stringifying the list of ids, it'll just be a string.) The tradeoff involved here is

• Either the client will have to send another PATCH request to the server to update comment ids but I don't think this is very practical since we're planning of supporting multiple clients now just our own.
• The other solution is internally the comment objects will have a parent id and type (when the request is made by the client) this can be an internal fixed property and whenever hydrus encounters an object with this property then internally it'll update the parent object's child field to add new child id and everything else should work without any issues.
  Personally, I think the second approach is the best way to move forward right now. Please feel free to discuss and add provide suggestions. @chrizandr @vddesai1871 @Mec-iS

[vddesai1871]

@xadahiya that's exactly what I was looking for, thanks.

Each comment will have a parent associated with it. This can easily be defined in the APIDoc and regarding the client, in any case, the client must know the article details it is adding the comment to, so it should be able to pass the parent ID while creating the comment object to be added. (I can't think of a scenario where someone will try to add a comment without knowing which article/issue it should be added. You can think of something please do let me know.)

I was considering the unRESTy case where the client can add new comments at article/id/comments so the Comment instance in the request might not require article_id field as it is already in the url. But as it's ruled out, the paragraph above makes perfect sense.

The IRI referencing the comments will use the article ID for example /Issues/3/comments, then hydrus will use the id 3 to generate a partialCollection to retrieve the comments with parent id 3. (The IRI generation, in this case, is really simple. It'll just be //)

If I am not missing something then it will require us to create a new resource named NestedCollection or something, mapped with api/<Collectiontype>/<ID>/<NestedCollectiontype> and having only get responder.

The other solution is internally the comment objects will have a parent id and type (when the request is made by the client) this can be an internal fixed property and whenever hydrus encounters an object with this property then internally it'll update the parent object's child field to add new child id and everything else should work without any issues.

Yes, it seems better than other approaches.

[xadahiya]

If I am not missing something then it will require us to create a new resource named NestedCollection or something, mapped with api/<Collectiontype>/<ID>/<NestedCollectiontype> and having only get responder.

Exactly!
You can then parse and use the child ids of item Collectiontype/<ID> to create a partial collection that will have the type of <NestedCollectiontype>