refactor: migrate from pip-tools to uv for dependency management

[TeaDrinkingProgrammer]

Replace the pip-tools based dependency management workflow with uv,
consolidating all dependencies into pyproject.toml and a single uv.lock file.
This simplifies and speeds up the development setup greatly.

Changes:

• Switch build backend from setuptools to hatchling
• Move all dependencies from requirements/*.in to pyproject.toml
• Remove setup.cfg in favour of .flake8 and pyproject.toml
• Remove Makefile in favour of poethepoet tasks
• Upgrade main python version (CI/CD, .python-version, etc.) to 1.12
• Use Ubuntu LTS latest for readthedocs.yaml
• Refactor and update Dockerfile to use uv for installation
• Upgrade the Debian version in the Dockerfile from bookworm to trixie
• Add .python-version for consistent Python version management
• Replace pip-tools with uv in all CI/CD workflows
• Remove legacy build and update scripts (to_pypi.sh, ci/update-packages.sh, ci/run_mypy.sh)
• Update documentation to reflect changes

Description

Fixes #1643. See commit for details.

• Added changelog item in documentation/changelog.rst

Look & Feel

N.A.

How to test

This is quite an extensive change, so these are the things I can think of:

• Running the CI/CD pipeline
• Running the poethepoet tasks
• Building the Dockerfile and running the Docker compose file.
• Running flexmeasures locally and setting up with UV

Further Improvements

Migrate from Flake8 to Ruff?

• Flake8 doesn't officially support pyproject.toml, so migration would consolidate all configuration in pyproject.toml
• Ruff is faster (written in Rust), supports pyproject.toml natively, and is the UV ecosystem's recommended linter
• Ruff can auto-fix many issues and is actively developed

Options:

• Current approach: Move the Flake8 config from setup.cfg to .flake8.
• Migrate to Ruff and consolidate config in pyproject.toml

Should we enable Dependabot for UV lockfile updates?

• GitHub Dependabot can monitor uv.lock for security updates
• Would create PRs automatically for dependency updates
• May generate frequent PRs depending on configuration

Options:

• Enable Dependabot with weekly updates
• Current approach (I think?). Keep manual updates only (status quo)
• Enable only for security updates

Migrate to using separate environments for plugins

Currently plugins use the same environment as the main application. This breaks the stability of the UV lock file.

I do not know enough about plugins to suggest how this can be done better.

6. Pre-commit Hook using UV

Use UV commands in the pre-commit hooks instead of separate pre-commit hook versions. See https://github.com/ElaadNL/openadr3-client/blob/main/.pre-commit-config.yaml

• Makes sure that the same version is always used in both the pre-commit hook and UV
• Only installs one version of the tool

Current approach: keep pre-commit hook versions, only migrate to poethepoet task when a python script was used before.

Compile bytecode

See https://docs.astral.sh/uv/guides/integration/docker/#compiling-bytecode

Current approach: compile bytecode in the Docker file.

Move to ranges in the dependency versions

Previously, dependencies were listed in the .in file, and 'compiled' to a requirements.txt file. In UV, it is customary to set a range (e.g. <=x.x.x), so that a UV lock upgrade always stays within minor versions and you need to manually update the major version. For now, I have applied that way of working, but this could be reverted.

Misc

• I removed to_pypy.sh, with the idea that the GH CD flow is enough, could be reverted though.

Related Items

• I noticed that the tests are fairly slow, I'll make a separate issue for that.

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[TeaDrinkingProgrammer]

I see I broke Sphinx in some way, I'll see if I can fix that

[nhoening]

Thanks!

Big change, but looking forward to the benefits in the CI pipeline and in development.
Also well done in cleaning files we don't need anymore and fixing smaller things like make commands everywhere!

1. It seems the dependencies are not kept per Python version anymore (just for 3.12)? So if somebody runs FlexMeasures on 3.10 and runs into a bug, how would we reproduce his exact environment? (This is not a question about this PR per se, more about the new way of working, probably. Sorry)
2. Now the dependencies all have a minimum version. Where did that come from?
3. We have Dependabot scanning requirements already. Maybe we can observe how it picks up uv.lock. I'd expect it would do well.
4. I have no inclination to stick with flake8, so if that is cleaner we can move to ruff. Any other thoughts on that @Flix6x ?
5. Your comment about plugins might have to do with some legacy approach we should clean up. (A) Any plugin can be added by path, and dependencies installed in a separate step. That is the MVP. (B) We also think it would be nice if a plugin could be installed to the venv and be referenced by its package name only. The cookiecutter template is outdated and needs to be cleaned. I recently moved fm-entsoe to pyproject.toml at least. So all existing plugins can continue to at least use version (A), and FlexMEasures will then use uv to install the plugin deoendencies if it finds them (or you are responsible to do that yourself actually). And we should later update the preferred way for plugins to do (B) (via uv).
6. I will take a look at poethepoet, I didn't know it yet. Not married to Makefiles, either!

[nhoening]

Cool.
We usually use HiGHs these days, any reason you used Cbc?

[TeaDrinkingProgrammer]

Some tests failed when I didn't have CBC installed

[nhoening]

Ah, maybe you have a different OS (Fedora?) than our team (I believe we have PopOS and MacOS) and that might come across unique problems :)

[TeaDrinkingProgrammer]

Yes that's right. For some reason Fedora chooses to name the binary 'Cbc' instead of 'cbc'. They're the only ones doing it as far as I know, so it's no wonder it flew under the radar.

[nhoening]

I see I broke Sphinx in some way, I'll see if I can fix that

Probably not, take a look at #1972

[TeaDrinkingProgrammer]

I see I broke Sphinx in some way, I'll see if I can fix that

Probably not, take a look at #1972

Ah, I see. I also added a fix. I'll revert it tomorrow and rebase

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #31491634 | 📁 Comparing ab00096 against latest (d2d93ce)

🔍 Preview build

Show files changed (56 files in total): 📝 55 modified | ➕ 0 added | ➖ 1 deleted

File	Status
changelog.html	📝 modified
configuration.html	📝 modified
index.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.assets.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.generic_schemas.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.search.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.sensor_data.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.sensors.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.users.html	📝 modified
_autosummary/flexmeasures.api.v3_0.assets.html	📝 modified
_autosummary/flexmeasures.api.v3_0.sensors.html	📝 modified
_autosummary/flexmeasures.api.v3_0.users.html	📝 modified
_autosummary/flexmeasures.data.models.charts.belief_charts.html	📝 modified
_autosummary/flexmeasures.data.models.data_sources.html	📝 modified
_autosummary/flexmeasures.data.models.forecasting.model_spec_factory.html	📝 modified
_autosummary/flexmeasures.data.models.forecasting.pipelines.base.html	📝 modified
_autosummary/flexmeasures.data.models.forecasting.pipelines.predict.html	📝 modified
_autosummary/flexmeasures.data.models.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.models.planning.html	📝 modified
_autosummary/flexmeasures.data.models.planning.storage.html	📝 modified
_autosummary/flexmeasures.data.models.planning.utils.html	📝 modified
_autosummary/flexmeasures.data.models.time_series.html	📝 modified
_autosummary/flexmeasures.data.schemas.account.html	📝 modified
_autosummary/flexmeasures.data.schemas.attributes.html	📝 modified
_autosummary/flexmeasures.data.schemas.forecasting.html	📝 modified
_autosummary/flexmeasures.data.schemas.forecasting.pipeline.html	📝 modified
_autosummary/flexmeasures.data.schemas.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.schemas.io.html	📝 modified
_autosummary/flexmeasures.data.schemas.reporting.aggregation.html	📝 modified
_autosummary/flexmeasures.data.schemas.reporting.html	📝 modified
_autosummary/flexmeasures.data.schemas.reporting.pandas_reporter.html	📝 modified
_autosummary/flexmeasures.data.schemas.reporting.profit.html	📝 modified
_autosummary/flexmeasures.data.schemas.scheduling.html	📝 modified
_autosummary/flexmeasures.data.schemas.scheduling.metadata.html	📝 modified
_autosummary/flexmeasures.data.schemas.scheduling.process.html	📝 modified
_autosummary/flexmeasures.data.schemas.sensors.html	📝 modified
_autosummary/flexmeasures.data.schemas.sources.html	📝 modified
_autosummary/flexmeasures.data.schemas.times.html	📝 modified
_autosummary/flexmeasures.data.schemas.utils.html	📝 modified
_autosummary/flexmeasures.data.services.forecasting.html	📝 modified
_autosummary/flexmeasures.data.services.sensors.html	📝 modified
_autosummary/flexmeasures.data.services.users.html	📝 modified
_autosummary/flexmeasures.ui.utils.breadcrumb_utils.html	📝 modified
_autosummary/flexmeasures.ui.utils.view_utils.html	📝 modified
api/v3_0.html	📝 modified
cli/change_log.html	📝 modified
concepts/commitments.html	➖ deleted
concepts/device_scheduler.html	📝 modified
dev/automated-deploy-via-GHActions.html	📝 modified
dev/ci.html	📝 modified
dev/dependency-management.html	📝 modified
dev/docker-compose.html	📝 modified
dev/setup-and-guidelines.html	📝 modified
host/data.html	📝 modified
plugin/customisation.html	📝 modified
tut/toy-example-setup.html	📝 modified

[Flix6x]

Ruff over flake8 sounds good. This PR already touches a lot, though. Does it make sense to do even more in one PR?

[Flix6x]

Is there something of a migration table of what the make commands have been replaced with? Or, even more backwards compatible, would it make sense to keep the old commands, but just have them run the new commands (with a deprecation warning if we want).

[TeaDrinkingProgrammer]

Ruff over flake8 sounds good. This PR already touches a lot, though. Does it make sense to do even more in one PR?

Yes I agree, I'll open a new issue

[TeaDrinkingProgrammer]

Is there something of a migration table of what the make commands have been replaced with? Or, even more backwards compatible, would it make sense to keep the old commands, but just have them run the new commands (with a deprecation warning if we want).

That's a good point, now the users workflow will break and they won't know what happened. I added a new Makefile that executes the equivalent command with a warning where it makes sense (like make install), otherwise it only displays a warning (like install-pip-tools)

[TeaDrinkingProgrammer]

Thanks!

Big change, but looking forward to the benefits in the CI pipeline and in development. Also well done in cleaning files we don't need anymore and fixing smaller things like make commands everywhere!

1. It seems the dependencies are not kept per Python version anymore (just for 3.12)? So if somebody runs FlexMeasures on 3.10 and runs into a bug, how would we reproduce his exact environment?  (This is not a question about this PR per se, more about the new way of working, probably. Sorry)

2. Now the dependencies all have a minimum version. Where did that come from?

3. We have Dependabot scanning requirements already. Maybe we can observe how it picks up uv.lock. I'd expect it would do well.

4. I have no inclination to stick with flake8, so if that is cleaner we can move to ruff. Any other thoughts on that @Flix6x ?

5. Your comment about plugins might have to do with some legacy approach we should clean up. (A) Any plugin can be added by path, and dependencies installed in a separate step. That is the MVP. (B) We also think it would be nice if a plugin could be installed to the venv and be referenced by its package name only. The cookiecutter template is outdated and needs to be cleaned. I recently moved [fm-entsoe](https://github.com/SeitaBV/flexmeasures-entsoe/blob/main/setup.py#L13) to pyproject.toml at least. So all existing plugins can continue to at least use version (A), and FlexMEasures will then use `uv` to install the plugin deoendencies if it finds them (or you are responsible to do that yourself actually). And we should later update the preferred way for plugins to do (B) (via `uv`).

6. I will take a look at poethepoet, I didn't know it yet. Not married to Makefiles, either!

Thanks 😄

1. No problem, I also had to look that up. UV makes sure all possible configurations are supported, so Python versions and architectures. As https://docs.astral.sh/uv/concepts/resolution/#universal-resolution describes:

During universal resolution, all required packages must be compatible with the entire range of requires-python declared in the pyproject.toml. For example, if a project's requires-python is >=3.8, resolution will fail if all versions of given dependency require Python 3.9 or later, since the dependency lacks a usable version for (e.g.) Python 3.8, the lower bound of the project's supported range. In other words, the project's requires-python must be a subset of the requires-python of all its dependencies.

2. That was one of the more opinionated changes. Your previous setup had one step in dependency upgrading (as I understand it), you go from 'package' to 'package==x.x.x'. UV by default adds a '>=x.x.x' constraint. In practice, there is not a huge difference, in theory it means that the lockfile version can never be below the latest version at the time of adding the dependency. It still does not enforce SEMVER, because python packages do not follow that (I was under that impression, turns out I was mixing up NPM and PEP semantics)

3. Great! We use Dependabot ourselves too, and it automatically detects UV in our experience.

4. I made a separate issue for that, would be nice to add as well. Like I said there, I don't mind making a start on it, but I don't know what rules you would like to use of course.

5. I don't completely follow, so is the approach I used correct, install the requirements.txt in the VENV using the pip compat mode of uv, correct? Otherwise, the documentation and docker compose needs to be updated.

6. I didn't know it before either but it works well and does everything you need it to do. The main idea is to move to Python-first tooling, so it is easier to work with.

[nhoening]

I want to test this, but have to find some time first.

• install uv
• create new venv (or equivalent) with it
• start and click around
• run tests
• build docker image & compose stack
• run the tutorial scripts against it
• look at GH actions

I am okay with fixing a few leftovers while we go along.

However, I want to release v0.31 before merging this PR, so that release process (documented in the TSC repo) will go smoothly.
That is hopefully on a few days.

[TeaDrinkingProgrammer]

Slightly improved the CI/CD: it was implicitly syncing some uv dependencies because of the invocation of the uv run poe script, so I took the change to move the pre-commit scripts to use local UV commands and the GH action to pre-commit-uv. The advantage to that is that we now per definition always us the same version of linting tools for everything with faster run times as a bonus.

One bug that came out of that though, is that it now uses the flake8-blind-except plugin correctly where it did not before. How would you like to deal with that?

[nhoening]

Slightly improved the CI/CD: it was implicitly syncing some uv dependencies because of the invocation of the uv run poe script, so I took the change to move the pre-commit scripts to use local UV commands and the GH action to pre-commit-uv. The advantage to that is that we now per definition always us the same version of linting tools for everything with faster run times as a bonus.

Maybe you answered my other comment here. How is it guaranteed that my local dev environment and GH Actions will use the same lint tool versions?

One bug that came out of that though, is that it now uses the flake8-blind-except plugin correctly where it did not before. How would you like to deal with that?

Oh, interesting.

1. How many blind excepts are there?
2. If we move away from flake8 soon, let's maybe not bother with this today. This PR is big enough as it is.

[TeaDrinkingProgrammer]

Slightly improved the CI/CD: it was implicitly syncing some uv dependencies because of the invocation of the uv run poe script, so I took the change to move the pre-commit scripts to use local UV commands and the GH action to pre-commit-uv. The advantage to that is that we now per definition always us the same version of linting tools for everything with faster run times as a bonus.

Maybe you answered my other comment here. How is it guaranteed that my local dev environment and GH Actions will use the same lint tool versions?

I'll explain it a bit more thoroughly:

Before:

Pre-commit installs itself using PIP, makes its own venv, and uses the flake8 etc. in that venv. Every dependency is installed by cloning the git repo, then running pip install . The version is determined using the rev field in the pre-commit.yaml file.

https://pre-commit.com/#python

Now:
First, we run uv sync --only-group dev --frozen. This sets up a VENV and installs all dependencies defined in the dev group into the VENV.

Because the pre-commit now just runs uv commands thanks to the language: system flag, these commands will use the versions installed by UV instead of the separately installed binaries through cloning the git repo.

Because this setup uses the same lock file, the version will always be the same as the versions you have installed locally.

Using uv-pre-commit means that pre-commit will install itself using UV too, but that is just a nice performance bonus, it doesn't change the running behaviour.

Another plus is that it doesn't have to clone the git repos any-more.

One bug that came out of that though, is that it now uses the flake8-blind-except plugin correctly where it did not before. How would you like to deal with that?

Oh, interesting.

1. How many  blind excepts are there?

2. If we move away from flake8 soon, let's maybe not bother with this today. This PR is big enough as it is.

There were 42 blind except errors. One error I forgot to mention is a whitespace error. It is not a big error, but I' ll give it the same treatment.

Good point, I've added a global ignore and an explanation for those errors for now

[nhoening]

Thanks for taking the time to explain more of this!

As I said earlier, I'd love to move on this soon, but this week we had other priorities and next week I'll be off work. Not sure if someone else can take the time until I'm back. Busy times. I hope it's okay that this sits around a bit longer.

Good to get v0.31 out first, also because there might be downstream effects. I can think of two.

First, cloudinfra scripts might need an update. We use internally ansible and terraform, and we might need to update a few places in the code and test on staging, not sure.

Second, plugins. You mentioned that you "install the requirements.txt in the VENV using the pip compat mode of uv". I still see only pip install (...) requirements.txt, e.g. in the docker-compose file or customization.rst. Should that not use uv, then? Or is this what the "pip compat" mode does? That pip can be used to install into /app/.venv?
Then, plugins should only provide a requirements.txt, and that is taken into account. As I said earlier, plugins should actually become simpler in thi regard and this is a good approach :)