Skip to content

Add standardized sources and targets definitions across dataset modules #1373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
amir9667 wants to merge 216 commits into
base: dev
Choose a base branch
from
Open

Add standardized sources and targets definitions across dataset modules #1373

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
216 commits
Select commit Hold shift + click to select a range
1ca6fbf
Introduce DatasetSources and DatasetTargets that are added to each Da…
ClaraBuettner Jul 8, 2025
a014745
Add sources and targets attributes to Dataset-Metadata Table
ClaraBuettner Jul 8, 2025
3ce2d1c
Add function to read sources and targets from the database
ClaraBuettner Jul 8, 2025
f1fb5ba
Add DatasetSources and DatasetTargets to Zensus Dataset
ClaraBuettner Jul 8, 2025
d4329fe
Access dataset sources and targets from attributes DatasetSources and…
ClaraBuettner Jul 8, 2025
7a53748
Fix load_sources_and_targets function
ClaraBuettner Jul 8, 2025
9f41011
Register datasets already in the metadata-table before all tasks are …
ClaraBuettner Jul 8, 2025
b77b35a
Use DatasetSources and DatasetTargets in Heat Supply dataset for dist…
ClaraBuettner Jul 8, 2025
176d0bd
Add function to export sources and targets of all datasets
ClaraBuettner Jul 9, 2025
aae1029
Split overly long url-strings
ClaraBuettner Jul 9, 2025
76b61af
Use DatasetSources and DatasetTarget in HeatSupply.individual_heating
ClaraBuettner Jul 9, 2025
5542e46
Adding DatasetSource/Targets for PowerPlant Before making final changes
amir9667 Jul 21, 2025
f7d0dfd
initial commit
amir9667 Jul 26, 2025
179f752
For HEavyduty
amir9667 Jul 27, 2025
7438f4e
Initial Commit
amir9667 Jul 27, 2025
24cf837
initial commit
amir9667 Jul 27, 2025
0b7df36
Adding DatasetSources/Targets for review in re_potential_areas._init_.py
mheshammenisy Jul 29, 2025
8bd1675
Adding DarasetSources/Targets for "gas_areas.py" for review
mheshammenisy Jul 29, 2025
269e897
Adding DatasetSources/Targets for "industrial_gas_demand.py" for review
mheshammenisy Jul 29, 2025
ed21ff2
Adding DatasetSources/Targets for "society_prognosis.py" for review
mheshammenisy Jul 29, 2025
b368fb2
Adding DatasetSources/Targets for "substation_voronoi.py" for review
mheshammenisy Jul 29, 2025
44d75c0
Adding DatasetSources/Targets for "data_bundle._init_.py" for review
mheshammenisy Jul 29, 2025
9c333c9
Adding DatasetSources/Targets for "heat_demand._init_,py" for review
mheshammenisy Jul 29, 2025
d1fd382
Adding DatasetSources/Targets for "heat_demand_timeseries._init_.py" …
mheshammenisy Jul 29, 2025
253848b
Merge branch 'features/#1283-fill-dataset-sources-and-targets' of htt…
mheshammenisy Jul 29, 2025
678aa75
Adding Sources & Targets for Era5 for initial review
mheshammenisy Aug 16, 2025
f378699
Revert "Adding Sources & Targets for Era5 for initial review"
mheshammenisy Aug 16, 2025
0afd323
Fix: sources URLs syntax error in data_bundle
mheshammenisy Aug 25, 2025
945f4c1
Updated Files so far
amir9667 Sep 8, 2025
d8df796
Merge branch 'features/#1283-fill-dataset-sources-and-targets' of htt…
amir9667 Sep 8, 2025
feaccd8
Fix: Syntax error in CHP
amir9667 Sep 8, 2025
c7a59eb
Fix: Identation Error in demandregio
amir9667 Sep 8, 2025
4392a0a
Fix: match_nuts3_bl ERROR
amir9667 Sep 10, 2025
752ba48
fix: clean up sources/targets attributes and references in GasAreas
mheshammenisy Sep 7, 2025
1c5033f
fix: clean up sources/targets attributes and references in Industrial…
mheshammenisy Sep 7, 2025
fb91b0d
fix: clean up sources/targets attributes and references in SocietyPro…
mheshammenisy Sep 7, 2025
a3561b3
fix: clean up sources/targets attributes and references in Substation…
mheshammenisy Sep 7, 2025
4e35d64
add: define sources and targets for Era5
mheshammenisy Sep 7, 2025
c51abb0
add: define sources and targets for EtragoSetup
mheshammenisy Sep 7, 2025
0e17a01
add: define sources and targets for Loadarea
mheshammenisy Sep 7, 2025
0581122
add: define sources and targets for OSMBuildingsStreets
mheshammenisy Sep 7, 2025
73ad1ee
add: define sources and targets for Osmtgmod
mheshammenisy Sep 7, 2025
ffa9394
add: define sources and targets for RenewableFeedin
mheshammenisy Sep 7, 2025
b21edc9
add: define sources and targets for Saltcavern
mheshammenisy Sep 7, 2025
6c71c2a
add: define sources and targets for ScenarioParameters
mheshammenisy Sep 7, 2025
39e1fcf
add: define sources and targets for Tyndp
mheshammenisy Sep 7, 2025
15ed5b0
Fixing the Error
amir9667 Sep 10, 2025
31be9de
Merge branch 'dev' into features/#1283-fill-dataset-sources-and-targets
amir9667 Sep 17, 2025
7b47f52
fix(zensus): correct paths to data_bundle_egon_data/zensus_population…
mheshammenisy Sep 30, 2025
120b024
fix: resolve sources/targets attr & var errors after merge from dev
mheshammenisy Oct 7, 2025
30420b6
fix: align sources/targets references and attributes
mheshammenisy Oct 12, 2025
ffdf9a9
fix: align sources/targets references and attributes
mheshammenisy Oct 12, 2025
592355d
Last updated Datasources and Tragets
mheshammenisy Sep 30, 2025
34587a1
add: define sources and targets for mastr.py
mheshammenisy Oct 18, 2025
5f77e80
add: define sources and targets for sanity_checks.py
mheshammenisy Oct 18, 2025
d243d60
add: define sources and targets for gas_grid.py
mheshammenisy Oct 18, 2025
1bc1158
add: define sources and targets for fill_etrago_gen.py
mheshammenisy Oct 18, 2025
e4088ef
add: define sources and targets for fix_ehv_subnetworks.py
mheshammenisy Oct 18, 2025
8f7a51c
fix(vg250): correct double 'vg250' directory path issue
mheshammenisy Oct 18, 2025
8dfcdbb
fix(industrial_sites): add missing target (schema.table)
mheshammenisy Oct 18, 2025
074ed76
chore: update dataset version numbers
mheshammenisy Oct 18, 2025
dc6d8f2
Merge resolved
amir9667 Oct 19, 2025
c220b2b
Merge resolved and updated versions
amir9667 Oct 19, 2025
9947612
updating version
mheshammenisy Oct 24, 2025
543cdfd
fix: sources/targets and attributes
mheshammenisy Oct 24, 2025
203db2e
fix(substation): use single-line execute_sql for CREATE/DROP to avoid…
mheshammenisy Oct 24, 2025
f9411c9
add defines sources and targets for electrical_neighbours
mheshammenisy Nov 2, 2025
0ada842
add defines sources and targets for heat_etrago
mheshammenisy Nov 2, 2025
76e73d2
add defines sources and targets for hydrogen_etrago
mheshammenisy Nov 2, 2025
061c452
fix sources and targets attributes bugs (heat_demand)
mheshammenisy Nov 2, 2025
7566dac
fix sources and targets attributes bugs (renewable_feedin)
mheshammenisy Nov 2, 2025
7195b94
fix sources and targets attributes bugs (society_prognosis)
mheshammenisy Nov 2, 2025
026d905
DSM and dlr are recently refactored and the errors of industry and os…
amir9667 Nov 2, 2025
87efa07
Merge branch 'features/#1283-fill-dataset-sources-and-targets' of htt…
amir9667 Nov 2, 2025
df5ece4
Fixing the errors
amir9667 Nov 3, 2025
8e02869
fixing errors
amir9667 Nov 7, 2025
e8a5534
fixing errors
amir9667 Nov 7, 2025
e786192
fix(society_prognosis): correct schema name bug in DB table references
mheshammenisy Nov 7, 2025
ac49e0f
fix(heat_demand_timeseries): fix sources/targets attributes and refer…
mheshammenisy Nov 7, 2025
3823bfe
Fixing the Errors
amir9667 Nov 9, 2025
ea9deed
Fixing the Errors
amir9667 Nov 9, 2025
9907199
docs(osm): update download() comment
mheshammenisy Nov 18, 2025
dcf2fa7
fix(zensus): adjust sources and targets definitions
mheshammenisy Nov 18, 2025
ece1831
fix(heat_demand_timeseries): correct SQL query syntax
mheshammenisy Nov 18, 2025
a5b3aaf
fix(era5): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
723fba4
fix(etrago_setup): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
3f1e25c
fix(gas_areas): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
72eb0b1
fix(industrial_gas_demand): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
e3dd033
fix(loadarea): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
808bb20
fix(osmtgmod): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
91a0ba3
fix(re_potential_areas): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
a4ecbaa
fix(renewable_feedin): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
5496ebd
fix(scenario_parameters): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
4981127
fix(substation_voronoi): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
0eb5b27
fix(tyndp): adjust sources and targets attributes
mheshammenisy Nov 18, 2025
1ac0ea3
fix: clean up sources/targets attributes and references in Mastr
mheshammenisy Nov 19, 2025
501e9a1
fix: clean up sources/targets attributes and references in SanityChecks
mheshammenisy Nov 19, 2025
572ea3b
add sources and targets for bus.py
mheshammenisy Nov 21, 2025
ee7fdab
add sources and targets for h2_grid.py
mheshammenisy Nov 21, 2025
a9abde0
add sources and targets for h2_to_ch4.py
mheshammenisy Nov 21, 2025
ae2e6e4
add sources and targets for power_to_h2.py
mheshammenisy Nov 21, 2025
dfd4a83
add sources and targets for storage.py
mheshammenisy Nov 21, 2025
dd7fe29
update sources and targets in __init__.py
mheshammenisy Nov 21, 2025
ac94f60
add sources and targets for power_to_heat.py
mheshammenisy Nov 21, 2025
d879d07
add sources and targets for hts_etrago.py
mheshammenisy Nov 21, 2025
71b8773
updating sources in __init__.py
mheshammenisy Nov 21, 2025
6218d3d
adjust sources and target attributes in gas_grid.py
mheshammenisy Nov 21, 2025
0da4f73
adjust sources and target attributes in fill_etrago_gen.py
mheshammenisy Nov 21, 2025
e80a70e
adjust sources and target attributes in electrical_neighbours.py
mheshammenisy Nov 21, 2025
90e50e4
Fixed errors
amir9667 Nov 22, 2025
1115617
updating souce and targets
amir9667 Nov 22, 2025
c2c2b75
Fixing the Errors
amir9667 Nov 23, 2025
78f7c69
fix: correct sources file path in electrical_neighbours
mheshammenisy Nov 23, 2025
be22eac
fix: sources and targets imports
mheshammenisy Nov 23, 2025
862b351
fix: use HeatEtrago DatasetSources/Targets in power_to_heat
mheshammenisy Nov 23, 2025
95c133d
fix: use DatasetSources/Targets for hydrogen_etrago modules
mheshammenisy Nov 23, 2025
31bc500
etrago_hydrogen config in hydrogen bus
mheshammenisy Nov 23, 2025
2621bed
Fixing the error and refactoring load sources
amir9667 Dec 7, 2025
34e9a9c
add sources and targets for ch4_prod.py
mheshammenisy Dec 10, 2025
c291ea0
add sources and targets for district_heating_areas.__init__.py
mheshammenisy Dec 10, 2025
d49562f
add sources and targets for electricity_demand_etrago.py
mheshammenisy Dec 10, 2025
f767ca3
update sources in heat_demand_timeseries.__init__.py
mheshammenisy Dec 10, 2025
6ede8cf
import sources and targets for daily.py
mheshammenisy Dec 10, 2025
c4f74bd
import sources and targets for idp_pool.py
mheshammenisy Dec 10, 2025
ca9cc06
import sources and targets for service_sector.py
mheshammenisy Dec 10, 2025
fd71dfb
update sources and targets attributes for power_to_heat.py
mheshammenisy Dec 10, 2025
273cb94
update importing sources and targets from __init__ in bus.py
mheshammenisy Dec 10, 2025
5dbeecf
update sources and targets import in h2_grid.py
mheshammenisy Dec 10, 2025
37b7e8e
update sources and targets import in power_to_h2.py
mheshammenisy Dec 10, 2025
5e65f2e
update sources and targets import in storage.py
mheshammenisy Dec 10, 2025
79f962b
import sources and targets in parameters.py
mheshammenisy Dec 10, 2025
82568c6
update sources and targets attributes for ch4_prod.py
mheshammenisy Dec 10, 2025
71ff817
fix path in electricity_demand_etrago.py
mheshammenisy Dec 11, 2025
bf598d3
fix sources in heat_demand_timeseries/__init__.py
mheshammenisy Dec 11, 2025
4b32bdc
fix sources and targets attributes in idp_pool.py
mheshammenisy Dec 11, 2025
9c3dfb8
fix import error dataset
mheshammenisy Dec 11, 2025
2cc01ae
fix imports in heat_demand_timeseries modules
mheshammenisy Dec 11, 2025
bad4315
fix sources bug in electricity_demand_etrago.py
mheshammenisy Dec 11, 2025
c96371d
fix syntax error in service_sector.py
mheshammenisy Dec 11, 2025
f5eeafb
fix: load sources/targets inside functions (bus.py)
mheshammenisy Dec 14, 2025
dbdcaeb
fix: load sources/targets inside functions (h2_grid.py)
mheshammenisy Dec 14, 2025
feb2d92
fix: load sources/targets inside functions (h2_to_ch4.py)
mheshammenisy Dec 14, 2025
01ad79d
fix: load sources/targets inside functions (power_to_h2.py)
mheshammenisy Dec 14, 2025
2c258e7
fix: load sources/targets inside functions (storage.py)
mheshammenisy Dec 14, 2025
5ccf00e
fix: load sources/targets inside functions (daily.py)
mheshammenisy Dec 16, 2025
a982f72
fix: load sources/targets inside functions (idp_pool.py)
mheshammenisy Dec 16, 2025
372e32e
fix: load sources/targets inside functions (service_sector.py)
mheshammenisy Dec 16, 2025
088d115
fix: load sources/targets inside functions (power_to_heat.py)
mheshammenisy Dec 16, 2025
2bc7e4d
fix: update sources (osmtgmod/__init__.py)
mheshammenisy Dec 16, 2025
e82089b
fix: import load_sources_and_targets from datasets (substation.py)
mheshammenisy Dec 16, 2025
5e3e270
Revert "Fixing the error and refactoring load sources"
amir9667 Dec 19, 2025
7565a93
fixing the error related to chp.insert
amir9667 Dec 21, 2025
a37add2
Fixing the errors and refactoring the remainings
amir9667 Dec 21, 2025
f649a33
Refactored
amir9667 Dec 21, 2025
1c16860
Refactored and fixed the airflow's error
amir9667 Dec 21, 2025
2d946a9
errors fixed
amir9667 Dec 21, 2025
5740e79
fixing of airflow's error
amir9667 Dec 21, 2025
7036bc4
resolving init files
amir9667 Dec 21, 2025
1537ba5
resolving airflow error
amir9667 Dec 21, 2025
14f744d
resolving airflow error
amir9667 Dec 21, 2025
ade73ee
same: fixing errors!
amir9667 Dec 21, 2025
c98dbc7
fix: restore dataset config for H2 grid input files
mheshammenisy Dec 21, 2025
6674259
fix: add sources and targets for mv_grid_districts
mheshammenisy Dec 21, 2025
1f9da1e
Fixing airflow error
amir9667 Dec 23, 2025
280ef29
Fixing the airflow error
amir9667 Dec 23, 2025
6107975
going back to initial version to see if the error is there or not
amir9667 Dec 23, 2025
e32b7d8
restoring because the error is due to the initial code itself
amir9667 Dec 23, 2025
ad8fc8b
refactoring the remainings based on load_source_target import
amir9667 Dec 25, 2025
b7aeeec
refactoring the remainings based on load_source_target import
amir9667 Dec 25, 2025
384e9ba
Fixing the airflow's error
amir9667 Dec 25, 2025
c9817fa
Fixing the airflow's error
amir9667 Dec 25, 2025
f1ba6c8
Fixing the airflow's error
amir9667 Dec 25, 2025
50e7daa
refactoring the remainings based on load_source_target import
amir9667 Dec 26, 2025
0da8361
refactoring the remainings of CHP based on load_source_target import
amir9667 Dec 26, 2025
36c4fe0
Fixing the airflow's error
amir9667 Dec 26, 2025
673cae0
Fixing the airflow's error
amir9667 Dec 26, 2025
5b29d37
Fixing the airflow's error
amir9667 Dec 26, 2025
7ee1d63
Back to previous version
amir9667 Dec 27, 2025
25f770c
fix(heat_demand): remove legacy configuration comments from __init__
mheshammenisy Dec 27, 2025
8e18fa0
fix(saltcavern): remove legacy configuration comments from __init__
mheshammenisy Dec 27, 2025
28a4033
fix(scenario_parameters): remove legacy configuration comments from _…
mheshammenisy Dec 27, 2025
986107d
fix(scenario_parameters): load sources and targets inside function
mheshammenisy Dec 27, 2025
833eace
fix(society_prognosis): remove legacy configuration comments
mheshammenisy Dec 27, 2025
97dec35
refactoring the remainings of storage based on load_source_target import
amir9667 Dec 30, 2025
818a7a2
Fixing the airflow's error
amir9667 Dec 30, 2025
a16038f
Fixing the airflow's error
amir9667 Dec 30, 2025
7f07bd1
vg250_mv_grid_districts: define DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
be0854b
zensus_mv_grid_districts: define DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
00f126d
zensus_vg250: define DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
9a56370
power_etrago/__init__: add DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
db2d671
match_ocgt: load DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
8fb9953
electricity_demand/__init__: add DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
0f39717
electricity_demand/temporal: load DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
9c9771f
cts_buildings: define DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
57c0081
hh_buildings: define DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
a01dfa5
hh_profiles: define DatasetSources and DatasetTargets
mheshammenisy Dec 27, 2025
df4494c
refactoring the remainings based on load_source_target import
amir9667 Dec 31, 2025
c8c96a2
fix: electricity demand temporal sources/targets access
mheshammenisy Dec 31, 2025
6389b7c
fix: SQL f-string syntax in heat demand time series
mheshammenisy Dec 31, 2025
e10fadb
refactoring the remainings based on load_source_target import
amir9667 Dec 31, 2025
3254437
chore: bump version in GasAreas
mheshammenisy Dec 31, 2025
bbbfe11
chore: bump version in HydrogenEtrago
mheshammenisy Dec 31, 2025
eeff242
fix: correct schema.table SQL formatting in hydrogen bus
mheshammenisy Dec 31, 2025
f4ccfb1
go back to dev version to spot the error
amir9667 Jan 1, 2026
83e5fcb
error fixing in airflow
amir9667 Jan 1, 2026
2970ab9
error fixing in airflow
amir9667 Jan 1, 2026
9ce1b2f
error fixing in airflow
amir9667 Jan 1, 2026
9654735
fix: clean up sources/targets references in GasAreas
mheshammenisy Jan 3, 2026
750db22
fix: update sources/targets definitions in hydrogen_etrago init
mheshammenisy Jan 3, 2026
60d1f78
fix: clean up sources/targets references in HydrogenBusEtrago
mheshammenisy Jan 3, 2026
6bf2d30
fix: clean up sources/targets references in HydrogenStorageEtrago
mheshammenisy Jan 3, 2026
35e92d3
refactor: define sources and targets for scenario capacities
mheshammenisy Jan 9, 2026
1ae8f72
fix: refactor storage module in hydrogen_etrago
mheshammenisy Jan 9, 2026
2a184c9
Refactoring the MIT charging infrustructure
amir9667 Jan 9, 2026
235b541
fixing airflow's error
amir9667 Jan 10, 2026
a93d043
Remove local .spyproject config files
amir9667 Jan 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,37 @@ docs/_build

# Mypy Cache
.mypy_cache/


# Local data and backups
/data_bundle*
/datasets/
/demandregio*
/pypsa_technology_data_egon_data.zip
/PyPSA-technology-data-94085a8/
/industrial_sites/
/openstreetmap/
/tyndp/
/vg250/
/bnetza_mastr/
*.orig
*.pid*
*.csv
*.sql
*.zip
*.yaml

# Python and IDE folders
/airflow/
/.spyproject/
/docker/

# Other
/hstore_Extension
et --hard ORIG_HEAD
git status

*__BACKUP*
*__BASE*
*__LOCAL*
*__REMOTE*
Loading
Loading