Understanding how to improve container support

[vinistock]

We have a few issues related to better container support, in particular in complex setups that aren't simply having the editor connect to a single container. For example, a container with multiple projects inside or a container-inside-container situation.

I want to centralize all information into a single place (this discussion) so that we can more easily see the full picture and start deriving concrete tasks from it. From all of the related issues, we need to understand what exactly is the shortcoming with the solution provided by VS Code for more complex container setups.

Related issues/prs: #2223, #3386, #2919, #3586

The current container support

Currently, container support for the Ruby LSP relies entirely on VS Code's container support. This is ideal for extensions because the editor itself handles everything - and it is in a better position to do so.

Connecting to a container means that:

1. Any processes spawned by the extension will automatically run in the container with no need for extra logic. That doesn't just mean booting the language server, but every external command we provide, which includes all test related functionality and the debugger client
2. All file system operations are also automatically redirected to the container through the vscode.workspace.fs API

Extensions do not have to be concerned at all with where they are running. The editor knows where it is running and can abstract away all of the complexity.

What we need to understand

Ideally, we can push for a better solution at the editor level, so that all languages and extensions are benefitted from more elaborate container support. If that's not possible or if the VS Code team is not open to making their integrations more complex, then we need to understand the level of effort involved in rolling out our own version of it.

For example, #3386 proposes customizing the command to launch the language server, but that solution is incomplete as other shell commands we try to run will not be executed within the container. Exactly the same scenario for #3586, which proposes a similar solution that only fixes test related commands.

Someone might then propose allowing all commands to be prefixed by something like dip, but none of these would solve the file system API problem and there's simply no way that we can provide rich editor features without the ability of reading a file on the extension side. When invoking all vscode.workspace.fs APIs, we would end up reading files in the user’s machine instead of the container.

Next concrete steps

1. Gather information from the community about what exactly is missing from VS Code's container support. Has this been reported to the VS Code team already? Are they open to support more complex setups? Let's begin by trying to push for a general solution that benefits all languages before trying to roll out any custom abstraction
2. If they are not open to it, we need to understand the level of effort involved in creating our own abstraction for this. Prefixing shell commands with something like dip is easy, but adding our own file system layer on top of VS Code's is definitely not. We would essentially be building the solution that should live in the editor ourselves.

Important note: we may still decide to not support certain use cases even after gathering more details. The Ruby LSP is an extension for language support, which is already complex enough without broadening its scope into dealing with containerized environments.

[Splines]

Just to report a bit about my experience.

🎈 I like VSCode to not connect to my Docker containers

For me, it's just my limited time and laziness that make me favor still running VSCode completely separate from my Docker containers.

See more details for this rationale

• The docs for developing inside a dev container and for the advanced configuration are good and I'm positive that with enough time & energy, I could migrate our complex setup to a dev container.
• It's just that it's a lot of things to do until you achieve a working setup: setup Git here inside Docker, with GPG key forwarding, SSH key forwarding, then assign correct linux users, mount your local files correctly, setup env variables correctly, then you also might want to persist your bash history, etc. Step by step this doesn't seem like not too much effort, but if there's one thing I know from working with Docker, it's that it frequently takes soo much time to figure things out. And due to the cycle where you always have to rebuild the entire container to see effects, this is very tedious (in the sense of time-consuming and frustrating) and error-prone (due to not always great error messages).
• So if you already have a completely working, and somewhat complex Docker setup that is even fine for production, I don't see the added value in redoing my entire setup just that VSCode can run inside a container.

Furthermore, I like browsing through my code and linting it, without having to start up Docker Desktop first. I usually modify and add some code here and there, then realize it'd be nice to interact with the app now, e.g. browse the UI or debug something. That's when I startup Docker Desktop in the background and launch a simple docker compose up -d, done.

🎈 I still need some processes to run inside the main app Docker container

But everything comes with advantages & disadvantages. In this case, well... we cannot use (without added effort) any IDE feature that needs our app's runtime:

• We cannot use the "server dependent features" of Ruby LSP since "Ruby LSP Rails failed to initialize server", obviously, since our setup only works inside a Docker container, but we run Ruby LSP & VSCode outside of it. They know nothing about Docker in our setup. I can live with that since all the other features Ruby LSP is offering are awesome enough for a full-fledged developer experience in my eyes (but I might be proven wrong ;)).
• More of a pain point: I cannot run tests via the Ruby LSP extension. In this gist, you can see how we work around this. Basically, we use the connorshea.vscode-ruby-test-adapter extension and tell it to run a custom Python script, which executes rspec in a Docker container (and replaces absolute paths to the test files by relative paths). It's not an ideal solution, but it works for now and additionally we run tests in the CI as well.
• Debugging is another thing. But just put the debug gem in your Gemfile, expose a port in your docker-compose.yml file and use that port in the VSCode launch.json. See our launch.json here. So that's not at all a problem when VSCode itself knows nothing about our Docker containers.
• ESLint that we use for JavaScript and HTML linting/formatting also doesn't need to know anything about our Docker container and is working fine.

So apart from running tests being a bit hacky, I'm fine with the setup; it has proven to work well during the last year. We even run Cypress tests inside a Docker container and forward the UI in WSL 😅 (where WSL nowadays is automatically WSLg, so X11 works out of the box).

🎈 So what?

Unfortunately, I don't know how to really derive helpful ideas to improve container support of VSCode / Ruby LSP. Here, you proposed the "socket solution" (see also #1782):

For more complex setups, like the one mentioned, I believe it ultimately boils down to the LSP supporting connecting through sockets #1782. The LSP has to run inside wherever Ruby and your dependencies are installed and where the code lives - since it needs to read it all from disk to index the declarations.

Yes, this could improve our setup (for usage of "server dependent features" and running tests). However, I would then be in a situation where I'd have to start Docker Desktop to get any linting in my IDE, which I would really dislike.

For my purpose, a hooking mechanism in Ruby LSP would be great, such that we could get rid of the Ruby Test Explorer extension (which doesn't yet support the native VSCode Test Explorer UI, see here, and which seems a bit abandoned). Such a hook could allow me to execute a custom script that executes the test inside a docker container however I like, then reports the results back. As inspiration, for reporting, the Test extension uses a custom_formatter.rb, see also my gist where I modify the path to the file to point to the memory-mapped path in the Docker container. But as I said, it's a bit "hacky".

[YurySolovyov]

Laziness is a weak argument, IMO. People who raise tickets related to container DX/ergonomics are likely actively working on a particular project, so they are running containers, and want to have a good experience, and this should the case to optimize for

[Splines]

so they are running containers

You probably wanted to say "are running dev containers" as in VSCode Dev Container, right? Since our app is containerized as well in Docker. It's just that the VSCode Server is not running inside our Docker containers.

Laziness is a weak argument

I agree :)
(but time-constraints I've presented in the section "See more details for this rationale" may be an argument)

and this should the case to optimize for

Sure, I mean if dev containers make the life of extension developers & users of the extension a lot easier, then this path should be prioritized. Just wanted to add the perspective of somebody who hasn't warmed to DevContainers yet and has a working containerized setup (see this repo).

[Splines]

Conceptually, I saw the scenarios "Editing your file in an editor (with linting)" and "Running your app inside a Docker container" as orthogonal. But of course, we expect more than just editing a file from an IDE these days, so the boundaries of those two scenarios get blurrier. Think of:

• Running your tests inside the app environment, but reporting back the results in your IDE (with the exact line that is failing).
• Getting lint results that require executing code inside the app-environment (not "just" static file analysis).
• Debugging code in the app-environment by means of your IDE (though as outlined in my original comment, this I can already do without dev containers).

So, yes I do see the advantages of running a VSCode Server inside a Docker container. But there's still disadvantages and inconveniences, otherwise we wouldn't have this discussion here.

[YurySolovyov]

No, I didn't mean VSCode dev containers, I meant ones people run via

docker compose up

Sure, most of these are db/queue/cache type containers, but some are actual app containers, in case of ruby, probably with Rails.

Having good DX for running inside these would be great.
Also, there are editors besides VSCode.

[Splines]

Sure, most of these are db/queue/cache type containers, but some are actual app containers, in case of ruby, probably with Rails.

Then, we probably misunderstood each other, since in our setup we also run docker compose up to start our app (in our case even Ruby on Rails). See for example this sentence in my original comment:

That's when I startup Docker Desktop in the background and launch a simple docker compose up -d, done.

---

Also, there are editors besides VSCode.

Sure, it's just that this discussion is also mainly around VSCode DevContainers. See the various links in vinistocks discussion comment.

[palkan]

@vinistock

customizing the command to launch the language server, but that solution is incomplete as other shell commands we try to run will not be executed within the container

Could you please tell a bit more about "other shell commands we try to run"? A test running command can be customized in a launch.json file; I believe, others, too. It's more of a developer's responsibility.

When invoking all vscode.workspace.fs APIs, we would end up reading files in the user’s machine instead of the container.

Currently, you can solve this by organizing a project's files such as they have the same paths locally and in a container (I'm using the${PWD}+workdir trick). Dependencies? You can either keep them in the project's directory and mount into the container or accept the tradeoff of not having them LSP-available.

---

I don't think Ruby LSP must explicitly deal with containerization. Just give the user more flexibility (like a custom server run path and custom anything else we need). Call it "experimental", so unexperienced users do not touch it accidentally.

[summera]

I'd love to see better general support for running Ruby LSP in a containerized environment outside of VSCode and devcontainers. I'm using Neovim and usually run everything in containers (including apps) with a docker compose file. If I'm missing some secret functionality, please let me know, but I have not found a way to run Ruby lsp in the container and connect to it remotely. I think it should be possible to use Neovim's LSP RPC functionality if it's possible to run Ruby LSP on a port within the container. Maybe related to #1782 ?

[palkan]

I have not found a way to run Ruby lsp in the container and connect to it remotely

I'm using Ruby LSP in a containerized env (with Zed) via Dip as follows:

• a custom bin/ruby-lsp executable that runs within a container: https://github.com/evilmartians/ruby-on-whales/blob/feat/lsp-support/template/ruby-lsp

• a Docker service configuration: https://github.com/evilmartians/ruby-on-whales/blob/c2d00fc29126cfa86594bef2a0a5ca28e43f1f4f/template/dip.yml.tt#L65-L69

• a PWD trick to have a project's path equal in and out of Docker: https://github.com/evilmartians/ruby-on-whales/blob/c2d00fc29126cfa86594bef2a0a5ca28e43f1f4f/template/compose.yml.tt#L16

Finally, you can configure Bundler to store deps in the vendor/bundle or .bundle/gems path and mount it to the same container, so Ruby LSP can see them (though, it can be slowish on Mac; I prefer to keep deps in a volume and I'm fine with not being able to open them in the editor; LSP still sees them, so annotations and such work).

[summera]

@palkan really appreciate the info! I'll check this out. Where is the custom ruby lsp executable coming from?

[palkan]

custom ruby lsp executable

it's just a bash wrapper over smth like docker compose run <service> bundle exec ruby-lsp; I need it to specify as a custom binary path for a Zed Ruby LSP plugin; with Neovim, I guess, you don't need a wrapper, you can just specify a custom command to launch an LSP server.

[Splines]

My ideal scenario would be this one:

• I connect to WSL (or remotely to a Linux server) that runs the VSCode server.
• For the files within WSL, Ruby LSP provides static code analysis & linting for me. This works even if I don't have any containers running.
• Additionally, if I have an app container running (which has the code from WSL mounted into it), Ruby LSP can spin up an extra instance of it inside this container. This is to provide runtime-aware analysis, debugging and test running. Ruby LSP could report back the result to the main Ruby LSP instance, which is running inside the remote VSCode, by means of a socket.

[Splines]

But of course I understand that VSCode dev containers have their own charm.

• (+) The Ruby LSP extension wouldn't have to implement the complex scenario presented above (running inside and outside of a Docker Container at the same time)
• (+) Developer onboarding is a lot faster since it's just spinning up a Docker container.
• (+) With Dev Containers you get a highly reproducible environment.
• (-) It requires a one-time effort by developers to get the Dev Container to work with a potentially very complex setup. Maybe some things cannot even be reflected in Dev Containers.
• (-) You have to start Docker (slow) to work with your code, otherwise you don't even get static code analysis (e.g. syntax highlighting).

[d-lebed]

Hi @vinistock, thank you for centralizing the discussion here.

I’d like to offer a slightly different perspective. It feels like we are focusing heavily on "how to fit everything into VS Code's native container support (Dev Containers)", whereas the reality for many teams involves complex docker compose setups that exist independently of the editor.

As users, we often cannot or do not want to migrate our entire workflow to VS Code Dev Containers. Furthermore, while the current focus is on the VS Code extension, ruby-lsp itself is a server that should ideally remain agnostic. Focusing too tightly on VS Code's internal remote architecture might limit the server's portability to other editors (Neovim, Emacs, Zed, etc.) that face similar challenges.

Fundamentally, running the LSP in a containerized environment (without Dev Containers) boils down to solving two distinct problems:

• Process Invocation: We need a way to launch the ruby-lsp process inside the container (e.g., passing custom env vars, using docker compose exec, etc.).
• Path Mapping: We need to map the file system paths seen by the editor (host) to the paths seen by the LSP server (container).

Regarding the concern about the File System API: In a typical docker compose setup, the source code is mounted via volumes. This means the files do exist on the user's machine. The editor can read/write them locally as usual. The only missing piece is the translation layer—telling the LSP that /Users/me/project/file.rb on the host corresponds to /app/file.rb inside the container.

We don't necessarily need to rebuild a full FS abstraction layer on top of VS Code. If we solve the Path Mapping (translating URIs between client and server) and the Execution Command, we solve the majority of these "complex setup" use cases without forcing full Dev Container adoption.

My PR #2919 attempted to address exactly these two points: allowing a custom launch command and handling the path mapping logic. I believe revisiting this "lighter" approach is crucial for supporting existing Docker workflows that cannot be replaced by VS Code Remote Containers.