Plugins: Support installing plugins from subpackages

[megannissel]

Some folks from TNC have developed an InVEST Calibration Assistant tool that works with AWY, SWY, SDR, and NDR. We've been talking with them about turning this tool into an InVEST plugin. Since the tool works with several InVEST models, but requires different inputs depending upon which model the user wants to calibrate, it may actually make more sense to develop this as a suite of plugins (one for each supported model). This would allow for a cleaner Workbench UI and less-complicated conditionality for inputs.

Right now, if the TNC folks wanted to develop a calibration plugin for each supported model, they would either need a separate GitHub repo for each plugin or would need separate branches for each plugin in the same repo (a bit hacky, but could possibly work?). This isn't ideal, since these plugins would likely share code under the hood. The branch-based solution would likely make development and maintenance more challenging.

Instead, it would be great if we could support installing plugins from subpackages. This would allow plugins of the same "family" to reuse shared code, and eliminate the need for separate GitHub repositories (and code duplication).

One possible way to enable this: in the Workbench plugin installation modal, could we add a field for "subpackage" where a user could point to a directory in a GitHub repo containing a pyproject.toml for a subpackage?

Taking this a step further, it would be cool if, eventually, a "suite" of plugins like this could be installed together, into the same environment.

Estimated time to complete: 1-3 days for scoping, possibly another 1-3 for implementation and testing (depending upon how we decide to proceed)

[davemfish]

Instead, it would be great if we could support installing plugins from subpackages. This would allow plugins of the same "family" to reuse shared code, and eliminate the need for separate GitHub repositories (and code duplication).

A compromise could be to develop all the shared resources as a library that each plugin depends on. It avoids code duplication but does not avoid the problem of managing a bunch of separate repositories.

I think it's a great idea to support subpackages though.

Taking this a step further, it would be cool if, eventually, a "suite" of plugins like this could be installed together, into the same environment.

This would be very cool and is worth striving for, I think. Otherwise the overhead of installing several plugins in sequence would be a large burden, for users and developers testing their plugin.

[megannissel]

I experimented with a very naive implementation, where I had two plugins as subpackages with some shared utilities. I managed to get installing them into the same environment and having them listed as individual plugins in the workbench to work, but so far I haven't managed to get them to launch. I also discovered an additional gotcha to this approach: Since the workbench viewed them as two different plugins, removing one also removed their shared micromamba environment, and subsequently trying to remove the second caused an error (since the micromamba environment it was trying to delete didn't exist). If we do want to support installing multiple plugins in the same environment, we'll need to carefully consider how to handle the shared environment. The easiest solution, at least initially, may be to require removing all plugins that share an env?

[emlys]

I agree it makes sense to support installing from an arbitrary path in the git repo, rather than requiring pyproject.toml to exist at the top level. As you say, that would allow multiple plugins to coexist in one repo, and it would also allow plugin developers freedom in structuring their repo (for instance, the plugin could be just one part of a larger project, which could contain non-python code and follow a different repo layout).

Whether those plugins are subpackages (in the python packaging sense) is a separate question. Currently, each plugin needs its own pyproject.toml, making it an installable top-level package. I don't see a great advantage to loosening that requirement. If someone just wants a way to group their related plugin packages together, they can use namespace packages, or simply a common prefix (reference).