[Repository Request] New GitHub repository named direct-query

[lezzago]

Are you requesting a new GitHub Repository within opensearch-project GitHub Organization?

Yes

GitHub Repository Proposal

opensearch-project/sql#4229

GitHub Repository Additional Information

1. What is the new GitHub repository name?
   direct-query
2. Project description and community value?
   The repository aim to store connections to other data sources like Prometheus and allow the ability to query the data sources as well as read and write metadata to the data sources such as reading/writing alerts in Prometheus.
3. What user problem are you trying to solve with this new repository?
   This creates the ability to have OpenSearch as a one stop shop to query and fetch data from multiple datasources and not just for OpenSearch. This is great because OpenSearch is optimized for log analytics and is not great for metric type data, so customers can query metrics from Prometheus all within OpenSearch Dashboards. This also lets users who have data in different data sources to see all of their data in a single pane view.
4. Why do we create a new repo at this time?
   Currently this direct query logic has been release in OpenSearch as experimental in the SQL plugin. It is necessary for the direct-query project to have its own repository for it to succeed and to prevent tech debt and unnecessary code in the SQL plugin.
5. Is there any existing projects that is similar to your proposal?
   Yes because there is experimental code in the SQL plugin that would be moved here.
6. Should this project be in OpenSearch Core/OpenSearch Dashboards Core? If no, why not?
   Or, shall we combine this project to an existing repo source code in opensearch-project GitHub Org?
   No because we dont want to have connecting to outside datasources to be tied to the core product and makes more sense as a plugin extension. There are no other plugin repos that would also make sense to store this code.
7. Is this project an OpenSearch/OpenSearch Dashboards plugin to be included as part of the OpenSearch release?
   Yes

GitHub Repository Owners

1. Who will be supporting this repo going forward?
   @lezzago @joshuali925 @goyamegh
2. What is your plan (including staffing) to be responsive to the community (at a minimum, this should include reviewing PRs, responding to issues, answering forum questions?)
3. Initial Maintainers List (max 3 users, provide GitHub aliases):
   @lezzago @joshuali925 @goyamegh

GitHub Repository Source Code / License / Libraries

1. Please provide the URL to the source code.
   https://github.com/opensearch-project/sql/tree/main/direct-query-core
   https://github.com/opensearch-project/sql/tree/main/direct-query
   https://github.com/opensearch-project/sql/tree/main/async-query-core
   https://github.com/opensearch-project/sql/tree/main/async-query
   https://github.com/opensearch-project/sql/tree/main/datasources
2. What is the license for the source code?
   Apache-2.0
3. Does the source code include any third-party code that is not compliant with the Apache License 2.0?
   No

What is the publication target(s)?

You can choose multiple targets from the list.

Maven Snapshots / Sonatype Nexus

Notes (DO NOT CHANGE)

Next Steps:

• If this is about creating a new GitHub Repository

  • Build Interest Group (BIG) and its members will review your proposal and provide feedback
    • Review of Proposal, asking questions, adding comments
    • If there is any concern regarding the naming / IP, additional IP review will be requested
    • Involve Subject Matter Experts from other repositories on the proposed topics
    • Ensure new repositories align with the foundation’s charter
    • Review the provided source code if any
    • Send final feedback and recommendations to the Technical Steering Committee
  • Technical Steering Committee (TSC) will have a vote based on BIG feedback, and reply back the vote as a comment in this issue by a TSC member
  • At least three positive (+1) TSC members' votes are necessary, and no vetoes (-1) after a one week period, then Admin Team will open a repo creation ticket with Linux Foundation
  • Linux Foundation verify the votes and create repo
  • Admin Team setup automations on repo settings, secrets, scanning, add initial maintainers, and more
  • Repository delivered to the original requester

• If you already have a GitHub repo and just want to add new publication target(s)

  • Admin Team will review your request and follow up

Track the progress of your request here: Engineering Effectiveness Board (view).
Member of @opensearch-project/admin will take a look at the request soon.
Thanks!

[peterzhuamazon]

Adding @msfroh @getsaurabh02 @bugmakerrrrrr @virajsanghvi to have a look at this and give their opinion.

As for infra, I want to know do you have any dependencies on other plugins?

Thanks.

[msfroh]

This is great because OpenSearch is optimized for log analytics and is not great for metric type data, so customers can query metrics from Prometheus all within OpenSearch Dashboards.

I disagree with the first part of this sentence, since opensearch-project/OpenSearch#19461 contradicts it. 😄 That said, encouraging folks to build metric UIs on OpenSearch Dashboards will be extremely useful once OpenSearch has an official metrics implementation.

I don't like the name direct-query, though. (I'm happy to see that @Swiddis seems to agree with me in opensearch-project/sql#4229 (comment).) In my opinion, it doesn't obviously describe what it accomplishes. Maybe external-query-connectors? (I don't know if I like that name, but I feel like it's still clearer.)

[virajsanghvi]

Wouldn't consumers want to be able to pick and choose data sources they support? Why have a single plugin for non-opensearch data sources vs one per data source? I just worry this doesn't scale over time with many data sources.

[msfroh]

Wouldn't consumers want to be able to pick and choose data sources they support? Why have a single plugin for non-opensearch data sources vs one per data source? I just worry this doesn't scale over time with many data sources.

In theory, it could be an extensible plugin. While it could initially ship with some set of "core" data sources, additional data sources could be added via SPI extensions. I guess the code is kind of pointing in that direction with the DataSourceClient and QueryHandler interfaces, though the existing factory is hard-coded for now.

[andrross]

Catch All Triage - 1 2