Add support to provide search capabilities for the stored artifacts

[hectorj2f]

With the increasing proliferation of new artifacts linked to manifests (e.g. sbom, scan reports, attestations, signatures...), there is a need to efficiently index the content of these artifacts to provide certain search capabilities. Obviously due to the heterogeneity of these artifacts, these search capabilities should be specific to each artifact type.

As an example, some of our use stories are:

• Get a list of images affected by a specific CVE .
• Get a list of images affected by a compromised signature.
• Find the latest scan report linked to a specific image digest.
• Get a list of images affected by a specific dependency.
• ...

Likewise, we are not sure that the usage of annotations for each artifact manifest would be enough to satisfy the aforementioned user stories.

[SteveLasker]

This would be a great example to leverage the distribution extension api, as the referres api does.
@hectorj2f, would you like to make a PR as a proposal?

Some other discussions on the topic:

• opencontainers/distribution-spec: Requirements: Search #71
• Common Registry Search and Eventing APIs

[hectorj2f]

@SteveLasker Yes, I'll have a look at the current examples and build something from it.

[michaelb990]

I see this capability as separate from the artifacts and references work. Happy to keep having this discussion here for now if it helps, but the eventual solution would probably be better suited to a separate project or working group focused around a search API that can be added on to a registry as an extension. We're not trying to define everything that can ever happen with artifacts, just the base primitives to allow for storing and simple discovery of artifacts that refer to images.

[SteveLasker]

Yes, to both.

[toddysm]

This ask was brought to my attention in several conversations and there is some interest in moving this forward. @michaelb990 proposal to create a WG sounds reasonable so I would like to bring this up in the next ORAS meeting for discussion. Let's use this issue in the next two weeks to gather interest for WG participation.

[rchincha]

Aye.

As a data point, this is how zot does it ... as a dist-spec extension,
https://github.com/project-zot/zot/blob/main/pkg/extensions/_zot.md
https://github.com/project-zot/zot/blob/main/pkg/extensions/search/search.md

[michaelb990]

Most of the open items in this repo have moved elsewhere -- I think we should do the same for this issue. The work that this project was created to do was folded into the OCI references working group here and is now part of the spec, pending the next release.

It looks like I don't have access to archive this repo, but I think it's time to do that.

[afflom]

Please sign me up for the working group.

For context, I've been involved with developing the Emporous (formerly UOR Framework) concept for the past 10 months which implements additions to the Image and Distribution Specs. Emporous also imports oras-go.

Prototype References:
Emporous Collection Spec (Draft) (Proposed change to Image Spec) - https://github.com/emporous/collection-spec

Emporous' approach:

• Typed attribute field on descriptors (Similar to annotations, but value is json.RawMessage/byte slice)
• Dynamic schema registration in the distribution server (Corresponds to typed attributes)
• Cross namespace linking

cc @jpower432 @sabre1041 @dmesser

[toddysm]

@michaelb990 I will be happy to bring the proposal to archive the artifact repo to the ORAS community in the next meeting and have the community decide at that time. The next meeting is on Jan 31st.

[toddysm]

The proposal that came up from the community meeting is as follows:

• Participants agreed to start the new WG under ORAS to collect and document scenarios
• Once the scenarios are considered complete, the WG participants will present the proposal for WG to OCI community
• Depending on the OCI response participants will make a decision if the specification should be implemented under OCI or will be started in ORAS.

See notes from Jan 31st 2023 in https://hackmd.io/P-O6n222TcSMoJgHmTTduw?both#Meeting-Notes

Please let me know if I misstated something - @jpower432 @sabre1041 @dmesser @afflom @rchincha @SteveLasker @AaronFriel @sabre1041 @FeynmanZhou @sajayantony @yizha1 @Wwwsylvia

I will work with @FeynmanZhou to create a repository for capturing the scenarios.

cc:// @michaelb990

[imjasonh]

• Participants agreed to start the new WG under ORAS to collect and document scenarios
• Once the scenarios are considered complete, the WG participants will present the proposal for WG to OCI community
• Depending on the OCI response participants will make a decision if the specification should be implemented under OCI or will be started in ORAS.

Why not have this discussion in OCI directly, from the beginning? Having a separate external group for requirements-gathering before an OCI WG can be approved and convened (whereby requirements will undoubtedly be gathered again, by a different group of people) just seems like unnecessary bureaucracy.

Or, going entirely the opposite direction: just define an extension without OCI's involvement or approval whatsoever, iterate there until it's perfect, and once it's adopted by clients and registries, propose it to OCI with a groundswell of support from happy users and registry operators.

Trying to do this work under OCI's umbrella while also pre-deciding most things about the API before involving OCI will probably only lead to headaches. (Ask me how I know! 🙃)

[imjasonh]

Also: how does this update square with the @michaelb990 's proposal in #119 ?

[michaelb990]

I've updated #119 to reflect the views of the community from the meeting notes. I think it should be ready to be merged once folks have a chance to take a look.

[toddysm]

I totally agree with you @imjasonh and both paths were discussed in the meeting. Though, the decision above was made collectively by the people who are interested to see such capability implemented in the registries.

I will let everyone else who was in the meeting chime in.

My personal position is that I would like to see this work done in OCI from the beginning. Though, my reluctance comes from the current state and discussions of OCI 1.1 (I believe this is what you meant😀). I am also not opposed to the extension approach. Though, I think that it fractures registry implementations and if it is brought to the OCI later on, we will still need to go through the same discussions and lengthy process. Once again, this is my personal position.

To be clear, the collective decision is to document the scenarios only and not do any work on specifying APIs or anything else.

I am not sure I follow how this issue is related to #119 except that #119 is the precedent for such work🙃