Vendorless Knative

[chizhg]

Currently, all the Knative projects are using go mod vendor, resulting in a huge number of files in the vendor directory.

Reasons for vendor dir removal

Maintenance burden for productivity

Having the vendor directory in every project creates a huge maintenance burden for productivity. Every single tool we use (various linters, compilers, formatters etc.) needs to be properly filtered, so it will give off false-positives. Here's a live example: knative-extensions/knobots#259 (there were multiple similar issues in the past)

Hard to count the contributions

Counting contributions is important for OSS projects, as it provides some insights into the health of the project. Having a vendor dir, impedes that, immensely. We need to filter out bots that contribute, and moreover, filter out the people contributions in vendor directory (or vendors in go.work is in place). Otherwise, the amount of the vendor code overwhelms any statistics.

Project looks dated

The enhancements in Go that made it possible to work without vendor directory safely has been made a long time ago - Go 1.13 released Sep 3, 2019 (over 3y ago). Since then, most of the code in Go has moved away from having the vendor directory. Sticking with it shows the project is old-fashioned, especially for newcomers. Remember when people attached external jar files to their Java projects, manually? That was before Maven was created to solve dependency management. The same was done for Go in 1.13+ releases.

Bad code habits

Having a vendor directory allows for relying on bad code habits, and hacks. Instead of using that just for Go code, we sometimes use it as a general tool for doing path traversal to some of the resources in the dependent projects. That's a smell, and may lead even to a vulnerability. Example of such smell can be seen in Knative-Serving:

$ find . -type l -printf '%p -> ' -exec readlink {} ';' | grep vendor
./config/core/configmaps/network.yaml -> ../../../vendor/knative.dev/networking/config/config-network.yaml
./config/core/300-imagecache.yaml -> ../../vendor/knative.dev/caching/config/image.yaml
./config/core/300-resources/certificate.yaml -> ../../../vendor/knative.dev/networking/config/certificate.yaml
./config/core/300-resources/domain-claim.yaml -> ../../../vendor/knative.dev/networking/config/domain-claim.yaml
./config/core/300-resources/ingress.yaml -> ../../../vendor/knative.dev/networking/config/ingress.yaml
./config/core/300-resources/serverlessservice.yaml -> ../../../vendor/knative.dev/networking/config/serverlessservice.yaml

(This is code smell, as symlinks are not cross-platform)

Status

The proposed roadmap has been presented at TOC (slides: https://docs.google.com/presentation/d/1eK7vhSRnahOq34WYkKJde8hvXMvb8eY3r76Wem5Ia9A) and was accepted.

Roadmap

Below is a list of issues that form this Epic.

1. First phase (the ordering is relevant)
   1. Hacks needs to be sourced without the vendor directory
      • Hacks needs to be sourced without the vendor directory hack#221
      • The knative.dev/hack go package shouldn't contain any go deps hack#318
   2. Knative hack scripts need to work with repos with vendor and without
      • Hack scripts shouldn't enforce vendor dir hack#310
      • update_licenses should still run when repos don't have vendor folder hack#315
   3. Notify the Knative contributors they can remove vendor if not using code-gen, or having symlinks to vendor/
      • Write documentation on Vendorless Knative hack#322
      • 🧹 Removing the vendor dir 🎆 knative-extensions/kn-plugin-event#307 (example PR)
      • 🧹 Removing the vendor dir 🎆 func#1966 (example PR)
      • Write an email, and Slack message.
   4. Check is it possible to execute K8s's shell scripts in vendorless project
      • Evaluate a possibility of calling codegen scripts in vendorless projects hack#385
      • 💝 Vendorless codegen via shell scripting hack#386
2. Second phase (could be done in any order)
   • Rewrite the Knative code-gen to a Go-native tool
     • Rewrite the Knative code-gen to a Go-native tool pkg#2809
     • PRs TBD
   • Rewrite the Kubernetes code-gen to a Go-native tool
     • kubernetes/code-generator#163
     • Set up an interim fork of the Kubernetes code-gen #180
     • PRs TBD
   • Resolve the symlink to vendor issue for Serving
     • Remove the symlinks to vendor directory serving#14326
     • PRs TBD

[dprotaso]

which can add a lot of noise when reviewing PRs that introduce extra dependencies.

If this is the sole reason I would split this into a separate commit and review the commits independently

Just as an FYI dropping this practice will be quite a lot of work since the majority reason we vendor deps is forpulling in non-go code artifacts

Some examples:

1. code-gen scripts from https://github.com/kubernetes/code-generator.
2. reconciler gen scripts in knative.dev/pkg that are used for generating injection clients.
3. knative.dev/hack scripts being propagated everywhere

Another extreme case is carrying patches for dependencies while we wait for them to be merged upstream. We've done that before for some K8s issues.

I actually do like seeing the vendored dependencies in PRs since it gives reviewers the ability to sanity check the changes being pulled in. This makes it easier to debug at times. If we drop vendoring I would hope we put an effort to improve our tooling around visualizing the changes - ie. see this dependabot PR - ko-build/ko#663

[chizhg]

If this is the sole reason I would split this into a separate commit and review the commits independently

That's true, but we don't have a way to enforce it and usually it's tedious (at least for me) to keep that in mind.

If we keep vendor/, I found the pull request file tree does help reduce the noise though.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[cardil]

/remove-lifecycle stale
/triage accepted