Features/#1274 unify and update to latest official open mastr #1369

CarlosEpia · 2025-10-19T08:13:09Z

Fixes #1274 .

Necessary changes to replace mastr data from 2024-01-08 for 2025-02-09. It was already tested for SH and DE. The results look consistent. Additional fixes related to power plants were also included here.

🧑‍💻 Contributor Checklist

Before requesting a review, make sure you've completed all of the following:

All tests pass locally or via CI
(for more information on local test, check tox in the Contributing section)
(CI tests are automatically executed when creating a PR, you can see the results of the checks below)
Workflow has run at least once in Test mode
(optional if no dataset changes are involved)
Relevant documentation is updated (API, new features, etc.)
Dataset-versions are updated when existing datasets are adjusted.
Added a note to CHANGELOG.rst about the changes
Added yourself to AUTHORS.rst

Optional:

Changes have been tested in Everything mode
Extend the checklist for reviewers: Which aspects should be reviewed in particular?

<!-- Example:
Please focus on validating the data handling in file XYZ.
-->

🔍 Reviewer Checklist

During your review, please check the following:

Is the code clean, readable, and efficient? Are there any oddities or obvious inefficiencies?
Does the code work as expected? (should already be verified by contributor)
Do all tests pass? (see CI results)
Is the documentation complete and up to date?
Is CHANGELOG.rst updated accordingly?
Is all necessary metadata complete and correct?
- If metadata is pending: Is there an appropriate issue filed?

📝 Additional Notes (optional)

…nto features/#1274-unify-and-update-to-latest-official-open_mastr

…icial-open_mastr

CarlosEpia · 2025-10-19T08:20:01Z

For eGon100RE no changes are seen in the etrago generators tables. But for eGon2035 there are differences vs the last run regarding biomass and CHPs, as can be seen in the table below that summarizes the total install capacities in DE:

I will go deeper in the code to find the reason of the differences and probably I will ask @ClaraBuettner about some methods in case of necessary.

…icial-open_mastr

ClaraBuettner · 2025-11-12T07:59:22Z

src/egon/data/datasets/power_plants/mastr.py

-        len_old = len(units)
-        ts = pd.Timestamp(
-            config.datasets()["mastr_new"]["status2023_date_max"]
-        )


Are you sure that we don't need that filter at all anymore?
I am just wondering if it would be needed if we want to create a status2024 scenario for example.

I see two different cases here:

Future scenarios: wind onshore and solar ground mounted generators benefit from the most updated available data, since those are the most possible locations for future generators.

Status scenario: We should, as you mentioned, filter the generators by commissioning date.

I think it makes sense to import all generators for futures scenarios, and filter the generators installed after the last day of our status scenario year using this function

Does it make sense for you?

Yes, I think we can do it like that.

Done in bf2593a 33b419e and ad3b399

ClaraBuettner · 2025-11-12T08:01:07Z

src/egon/data/datasets/power_plants/wind_offshore.py

+                "Cloppenburg": "50643382",
            }
-        elif ("220101" in osm_year) | ("240101" in osm_year):
+        elif "250101" in osm_year:


This means that the branch has to be combined with the osm update, right?

Okay, so that means that this branch should be merged after the osm, update, right? Is that already merged?

So far I have ran always the two branches together. To make it more clear, I merged the two of them 87dd6fa and will close both issues #1363 and #1274 by merging only this PR.

ClaraBuettner · 2025-11-12T08:01:55Z

src/egon/data/datasets/power_plants/wind_offshore.py

+                "Garrel/Ost": "23837631",
+                "Rastede": "23837631",
+                "Emden/Borßum": "34835258",
+                "Cloppenburg": "50643382",


Why did you move this? Is it the idea that it works for osm data from 2020 and 2025?

Some clarifications:

id_bus: Contains the osm_id that are common when using OSM data from 2020 and 2025

id_bus2: Contains all osm_id that changes between the 2 mentions OSM years

The in comparison with 2024, 5 substations change the osm_id and they had to be moved to id_bus2. I decided to do this to keep the compatibility with our last osm year, but I could adapt everything so we set that only data from osm2025 is suitable. Do you prefer to keep it like it is or change to work only with 2025?

ClaraBuettner · 2025-11-12T08:04:41Z

For eGon100RE no changes are seen in the etrago generators tables. But for eGon2035 there are differences vs the last run regarding biomass and CHPs, as can be seen in the table below that summarizes the total install capacities in DE: ... I will go deeper in the code to find the reason of the differences and probably I will ask @ClaraBuettner about some methods in case of necessary.

@CarlosEpia Could you please leave a comment if this is still up to date or not? I would just like to make sure that we are not confused later on when we have a look at merged PRs.

ClaraBuettner

Please take a look at the comments :)

ClaraBuettner · 2025-11-12T08:05:43Z

How is this PR related to #1368?

CarlosEpia · 2025-11-13T08:33:23Z

For eGon100RE no changes are seen in the etrago generators tables. But for eGon2035 there are differences vs the last run regarding biomass and CHPs, as can be seen in the table below that summarizes the total install capacities in DE: ... I will go deeper in the code to find the reason of the differences and probably I will ask @ClaraBuettner about some methods in case of necessary.

@CarlosEpia Could you please leave a comment if this is still up to date or not? I would just like to make sure that we are not confused later on when we have a look at merged PRs.

The cause of the differences was basically that the function scale_prox2now was working wrongly, which produces different results for different MASTR versions. Fixed here: 389a367

CarlosEpia · 2025-11-13T08:35:36Z

How is this PR related to #1368?

At least in my case, I've only seen the problem described in #1368 happening when using the MASTR data from February 2025.

ClaraBuettner · 2026-01-14T09:12:48Z

How is this PR related to #1368?

At least in my case, I've only seen the problem described in #1368 happening when using the MASTR data from February 2025.

Okay, so is it related to this PR, or was there already some progress with the problem described there?
I am just wondering if we need to adjust sth. here.

…1274-unify-and-update-to-latest-official-open_mastr

…icial-open_mastr

CarlosEpia added 23 commits September 25, 2025 08:42

change hydro and biomass to new MASTR

dc887d0

change other power plants to new MASTR

9aef960

change conventional generators to new MASTR

aa8546a

change gas generators to new MASTR

4610e9f

delete other references to MASTR_OLD

618f471

use MASTR_NEW

7abc367

use MASTR_NEW for CHPs

bb19abd

discard download of MASTR_OLD

e865c6f

Merge branch 'features/#1352-Make-egon-data-runnable-in-python3.10' i…

7f1b2a0

…nto features/#1274-unify-and-update-to-latest-official-open_mastr

use mastr data from 2025-02-09

810e822

update mastr version

a21bd30

use isort

b3c52ba

include documentation of fuction extract_and_preprocess_mastr

b652ce4

remove unnecessary columns

d340599

update version

407f699

ensure data types

82729c3

include again technologies for metadata

c97b85e

add function to download mastr_geocoding

97be7c7

Merge branch 'dev' into features/#1274-unify-and-update-to-latest-off…

2ff02c8

…icial-open_mastr

use OSM data from 01-01-2025

424523b

include 2025 in list of OSM validated data for wind offshore

f7a64fe

include all generators regardless start of operation

faee031

fix problem with nan values

2ae9efa

CarlosEpia marked this pull request as draft October 21, 2025 07:05

CarlosEpia added 5 commits October 21, 2025 10:40

fix function scale_prox2now

389a367

Merge branch 'dev' into features/#1274-unify-and-update-to-latest-off…

044e6fe

…icial-open_mastr

update osm_ids to match new OSM year

7adb0c2

relocate delete statement

bdff25e

update version

bc592f7

CarlosEpia added 7 commits November 4, 2025 08:15

make wind offshore compatible with OSM 250101

8f5343f

Merge branch 'dev' into features/#1274-unify-and-update-to-latest-off…

66b4634

…icial-open_mastr

update changelog

589125f

update chp dataset version

47781be

update chp storages version

59e3ca9

using Black

2cb0671

disable pylint warning for extract_and_preprocess_mastr

1d52cc9

CarlosEpia requested a review from ClaraBuettner November 10, 2025 08:19

CarlosEpia self-assigned this Nov 10, 2025

CarlosEpia marked this pull request as ready for review November 11, 2025 10:18

ClaraBuettner reviewed Nov 12, 2025

View reviewed changes

ClaraBuettner requested changes Nov 12, 2025

View reviewed changes

CarlosEpia added 5 commits January 16, 2026 14:54

Merge branch 'features/#1363-use-more-updated-OSM-data' into features/#…

87dd6fa

…1274-unify-and-update-to-latest-official-open_mastr

adapt function to work with English and German columns

ad3b399

use discard_not_available_generators in conventional gen

33b419e

Merge branch 'dev' into features/#1274-unify-and-update-to-latest-off…

7d0b7b4

…icial-open_mastr

use discard_not_available_generators for all other carriers

bf2593a

Features/#1274 unify and update to latest official open mastr #1369

Are you sure you want to change the base?

Features/#1274 unify and update to latest official open mastr #1369

Uh oh!

Conversation

CarlosEpia commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧑‍💻 Contributor Checklist

🔍 Reviewer Checklist

📝 Additional Notes (optional)

Uh oh!

CarlosEpia commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ClaraBuettner commented Nov 12, 2025

Uh oh!

ClaraBuettner left a comment

Choose a reason for hiding this comment

Uh oh!

ClaraBuettner commented Nov 12, 2025

Uh oh!

CarlosEpia commented Nov 13, 2025

Uh oh!

CarlosEpia commented Nov 13, 2025

Uh oh!

ClaraBuettner commented Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CarlosEpia commented Oct 19, 2025 •

edited

Loading

CarlosEpia commented Oct 19, 2025 •

edited

Loading