Adding arbitrary git remotes as code hosts

[imax9000]

I'm looking at the https://docs.sourcebot.dev/docs/connections/overview and don't see how to add a repository that I can fetch with git clone, but that's not hosted on one of the listed code host kinds.

Is there a way to add any repo that you can clone? If not, there probably should be an option to do that.

[msukkari]

Hey Max, we had this functionality in v2 but no longer support it in v3. If you could describe your use case I can see if we can add it back

[imax9000]

Umm, the use case is pretty obvious: adding any git repo regardless of how it's hosted, e.g.:

• https://git.kernel.org/
• https://go.googlesource.com/go
• https://savannah.gnu.org/
• https://cgit.freedesktop.org/ (deprecated in favour of a Gitlab instance, but that wasn't always the case)
• Any of the repos people self-host with Forgejo
• Any of the repos people self-host with a plain HTTP server
• Any of the repos hosted on https://tangled.sh
• Repos on my local machine, some of which aren't published anywhere

At various points in time I had all of the above (except maybe for https://tangled.sh) in my old Sourcegraph instances.

[brendan-kellam]

Will look into this. The approach I'm thinking is this: #250 (comment)

[brendan-kellam]

Yea I'm planning on supporting the local recursive case w/ glob patterns for the first version. This won't be supported for remote paths ofc.

I'm not familiar with that Sourcegraph feature, but it kind of sounds to me like a "templating" system that could be useful for when you have many remote repositories. Something like:

"my-connection": {
    "type": "git",
    "url": "https://github.com/sourcebot-dev/{{repo}}",
    "variables": {
        // each of these is substituted into {{repo}} to get
        // full remote path.
        "repo": [
            "sourcebot",
            "zoekt"
            // ... etc ...
        ]
    }
}

I'd say this one is out of scope for now, but curious to get your thoughts.

[imax9000]

Here are some examples I had in my Sourcegraph:

{
  "url": "git://git.kernel.org/pub/scm/",
  "repos": ["libs/klibc/klibc"]
}

{
  "url": "https://go.googlesource.com",
  "repos": [
    "go",
    "time",
  ]
}

As you can see, there isn't much to it: just a base URL and a list of suffixes to point to individual repos. I think stuff like fetch/update rate limits applies on per-code host basis, so grouping together repos also helps to not put excess strain on those hosts.

Your example does achieve what I have in mind, but I don't think a full-blown templating is even needed here. Having more than one variable doesn't make much sense (what should Sourcebot do then? Compute Cartesian product?), so it would be less confusing and error-prone if the configuration structure wouldn't allow adding more variables.

[msukkari]

You can specify a list of repos in a connection, and Sourcebot will fetch all of those repos from the code host provided. Is that the functionality you're describing here, or is it more involved?

[brendan-kellam]

@imax9000 We shipped support for this in https://github.com/sourcebot-dev/sourcebot/releases/tag/v3.2.1

I like your idea of having a repos prop - simpler than what I was suggesting with the template system. I will leave that as a future enhancement (or if you'd like, feel free to open a PR 😄 )

[imax9000]

Awesome, thank you!