chore: centralize proto files

[whynowy]

numaproj/numaflow#1402

[vigith]

Should we completely delete the proto file from here or just indicate in the CI process that it has diverged? Otherwise developing new changes might get tricker as it pulls from the main branch of numaflow.

[whynowy]

Should we completely delete the proto file from here or just indicate in the CI process that it has diverged? Otherwise developing new changes might get tricker as it pulls from the main branch of numaflow.

Good question.

I believe this would be the right process for any other sdks other than golang - however golang will always be the 1st one if there's any change to the protocols, we need to think about how to make it smooth for golang.

[whynowy]

I imagine the process of introducing proto file changes would be:

1. A PR in numaflow repo for proto file changes.
2. During Go sdk development, temporarily points to the numaflow PR branch to download proto files for codegen, and raise a PR;
3. Merge numaflow PR;
4. Merge Go sdk PR.
5. Work on other SDKs implementation.
6. Add a non-blocking CI to run make proto for all the sdks, to indicate if there's drift on the codegen.

Another option is, use Go SDK repo as the the location for centralized proto files.

@vigith - WDYT?

[vigith]

I imagine the process of introducing proto file changes would be:

1. A PR in numaflow repo for proto file changes.
2. During Go sdk development, temporarily points to the numaflow PR branch to download proto files for codegen, and raise a PR;
3. Merge numaflow PR;
4. Merge Go sdk PR.
5. Work on other SDKs implementation.
6. Add a non-blocking CI to run make proto for all the sdks, to indicate if there's drift on the codegen.

Sounds good to me.

I think we can make the main in this URL path can be overrideable via ENV var? This will help in fetching the proto file easier for development without editing the script.

[whynowy]

I imagine the process of introducing proto file changes would be:

1. A PR in numaflow repo for proto file changes.
2. During Go sdk development, temporarily points to the numaflow PR branch to download proto files for codegen, and raise a PR;
3. Merge numaflow PR;
4. Merge Go sdk PR.
5. Work on other SDKs implementation.
6. Add a non-blocking CI to run make proto for all the sdks, to indicate if there's drift on the codegen.

Sounds good to me.

I think we can make the main in this URL path can be overrideable via ENV var? This will help in fetching the proto file easier for development without editing the script.

Makes sense.

[yhl25]

Could we please hold off on merging this pull request for now? I am currently making some changes to the proto files in two of my pending PRs. Merging this right now could create conflicts with those changes.

[whynowy]

Could we please hold off on merging this pull request for now? I am currently making some changes to the proto files in two of my pending PRs. Merging this right now could create conflicts with those changes.

@yhl25 - can you remove the change request?

[whynowy]

Proto Generate will be green after numaproj/numaflow#1466 is merged.