Regrid updates and output fixes #447

sol1105 · 2025-08-13T14:59:44Z

Pull Request Checklist:

This PR addresses an already opened issue (for bug fixes / features)
- This PR fixes issue Problems to remap certain CORDEX, CMIP5/6 datasets provided to C3S #341 (together with already merged PR Regridding masks, robustness of dataset_utils functions and bug fixes #342)
Tests for the changes have been added (for bug fixes / features)
Documentation has been added / updated (for bug fixes / features)
HISTORY.rst has been updated (with summary of main changes)

What kind of change does this PR introduce?:

Fixed regrid.ipynb
Removed need for psymaps / psyplot dependency
utils/common.py:
- Updated require_module to accept an interval of unsupported versions
utils/dataset_utils.py:
- Added option to get_coord_by_type to suppress warning about a main variable not being identifiable
utils/output_utils.py:
- Added functions fix_netcdf_attrs_encoding and '_fix_str_encoding to fix utf-8 encoding issues for global and variable attributes
ops/base_operation.py:
- Applying above fix to correct utf-8 encoding issues
core/regrid.py:
- xarray version warning correctly triggered for versions outside of compatible version interval
- Grid.detect_extent:
  - determins now also extent in latitude
  - using now average resolution in x and y direction, respectively, rather than the average resolution, to determine the extent in lon/lat direction
  - now returning (lon_extent, lat_extent) instead of just lon_extent
- Grid._grid_from_ds_adaptive:
  - Added fix for regional grids that include either meridian or antimeridian
  - Grids that are considered global in lon or lat direction, will now be defined within 0,360 / -180,180 or -90,90 degrees_east/north
- Grid.to_netcdf:
  - Added fix for ds.encoding['unlimited_dims'] not being updated by xarray when dropping all time-dependent variables to write out solely the horizontal grid
  - Applying above fix for utf-8 encoding issues
- Grid.extent_lat, Grid.extent_lon: added new attributes
- Weights:
  - Will now use post_mask_source='domain_edge' setting when remapping a regional grid via nearest neighbour method, to avoid extrapolation beyond the original source domain (xESMF PR yet to be approved and merged)
Tests:
- Added / modified tests for code changes
- Added some tests to increase coverage
- Reverted skipping or xfailing tests that used to fail with engine="h5netcdf"

Does this PR introduce a breaking change?:

clisops.core.regrid.Grid.detect_extent now returns the tuple (lon_extent, lat_extent) rather than just lon_extent
Grid.extent now carries the combined extent in lon and lat direction and no longer just the extent in lon direction. If the grid is considered global in extent of both, lon and lat coordinates, Grid.extent is set to "global" else, it is set to "regional". Grid.extent_lon and Grid.extent_lat allow accessing the extent in only lon and lat direction, respectively.

Other information:

I can update HISTORY.rst once all changes made by the PR are approved and potential feedback worked in.
There are currently two xesmf PRs open (#444 #445) that should be merged and a xesmf release triggered before merging this in.

- utils/common.py: - Updated require_module to accept an interval of unsupported versions - utils/dataset_utils.py: - Added option to get_coord_by_type to suppress warning about a main variable not being identifiable - utils/output_utils.py - Added functions 'fix_netcdf_attrs_encoding' and '_fix_str_encoding' to fix utf-8 encoding issues for global and variable attributes - ops/base_operation.py: - Applying above fix to correct utf-8 encoding issues - core/regrid.py: - xarray version warning correctly triggered for versions outside of compatible version interval - Grid.detect_extent: + determins now also extent in latitude + using now average resolution in x and y direction, respectively, rather than the average resolution, to determine the extent in lon/lat direction + now returning (lon_extent, lat_extent) instead of just lon_extent - Grid._grid_from_ds_adaptive: + Added fix for regional grids that include either meridian or antimeridian - Grid.to_netcdf: + Added fix for ds.encoding['unlimited_dims'] not being updated by xarray when dropping all time-dependent variables to write out solely the horizontal grid + Applying above fix for utf-8 encoding issues - Grid.extent_lat: added new attribute - Weights: + Will now use "post_mask_source='domain_edge'" setting when remapping a regional grid via nearest neighbour method, to avoid extrapolation beyond the original source domain - Tests: - Added / modified tests for code changes - Reverted skipping or xfailing tests that used to fail with engine h5netcdf

coveralls · 2025-08-13T15:08:59Z

Pull Request Test Coverage Report for Build 19869800153

Details

76 of 95 (80.0%) changed or added relevant lines in 5 files are covered.
3 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.8%) to 87.288%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
clisops/utils/common.py	3	5	60.0%
clisops/utils/output_utils.py	21	26	80.77%
clisops/core/regrid.py	39	51	76.47%

Files with Coverage Reduction	New Missed Lines	%
clisops/utils/common.py	3	79.1%

Totals
Change from base Build 19505413673:	0.8%
Covered Lines:	3186
Relevant Lines:	3650

💛 - Coveralls

- Consistent distinction between extent, extent_lon, extent_lat - Adaptive grids with global extent in lon/lat will not exceed -180,180 or 0,360 in lon / -90,90 in lat - Modified tests accordingly - Added tests for yet untested parts of the regrid code to boost coverage

sol1105 · 2025-09-09T05:14:25Z

ToDo: Replace all .item calls with .data or .values (#451).

Zeitsperre · 2025-11-21T17:10:41Z

@sol1105 Is this ready for review? I can start looking at it next week if so.

sol1105 · 2025-11-21T17:51:56Z

@Zeitsperre Thank you, it would be great if you had a look the changes :)

The changes require a feature recently merged into the xesmf master branch. So we need to wait for a xesmf release (which is likely to happen pretty soon) before merging this.

Zeitsperre

LGTM. Given that @aulemahal implemented some of these changes in xesmf, I'll throw the ball to him to weigh in on this as well.

clisops/utils/common.py

clisops/utils/dataset_utils.py

clisops/utils/output_utils.py

Zeitsperre · 2025-11-24T16:22:56Z

clisops/utils/output_utils.py

+    s : str, byte
+        The string to be fixed. If the input is not of type str or bytes,
+        it is returned as is.
+    encoding : str, optional


optional suggests that encoding can be passed as None. Be sure to handle that case if so.

def _fix_str_encoding(s, encoding="utf-8"): [...] encoding : str, optional The encoding to be used. Default is "utf-8".

I thought optional only means that else the default is used and if None should be allowed it would be set as
encoding: str or None, optional?

From my understanding, optional and something | None are the same thing. It's a shorthand that resembles the optional you would find in other languages (like C++ or Rust) meaning that the function accepts a Null value.

For more info: https://docs.python.org/3/library/typing.html#typing.Optional

I think the confusion here stems from the caveat in the standard library: Note that this is not the same concept as an optional argument, which is one that has a default.

When I see ,optional in a docstring, I think of the call signature Optional. It's a bit ambiguous, I suppose.

clisops/utils/output_utils.py

environment.yml

tests/test_ops_regrid.py

for more information, see https://pre-commit.ci

sol1105 · 2025-11-25T15:16:40Z

@Zeitsperre Thanks for having such a thorough look at the changes :)

…regards to xesmf

sol1105 · 2025-11-28T19:59:23Z

@aulemahal Many thanks for all the effort you put into getting the xesmf release out 😃

@Zeitsperre I think I made all necessary changes and all tests pass. Only for Anaconda python3.10 the environment cannot be built because xesmf does no longer support python 3.10.

aulemahal · 2025-11-28T20:08:50Z

Sorry about the drop. I hope this is not problematic on your side ? I chose to drop support in the lack of resources to fix bugs related to older python/xarray/sparse versions.

Unless you have objections, I will try to remove xESMF 0.9 and 0.9.1 from PyPi because their "support" of Python 3.10 is only partial.

Zeitsperre · 2025-11-28T20:47:26Z

@sol1105 I think we can either drop Python3.10 (loses security support in a year; https://endoflife.date/python) or we can add a version check for the regridding tools, maybe something like:

Add a version specifier for xesmf: xesmf >=0.9.2; python_version >= '3.11'"
Add a check when someone tries to import the regridding functions that fails if python <= 3.10.

I would be fine with us dropping Python3.10; There have been many changes to clisops lately, and I still would like to see us split off core, possibly early 2026.

sol1105 · 2025-12-02T11:42:21Z

I think we can either drop Python3.10 (loses security support in a year; https://endoflife.date/python) or we can add a version check for the regridding tools

@Zeitsperre I do not have any preference here. Should these changes however be done in a different PR?

@cehbrecht What do you think?

cehbrecht · 2025-12-02T13:57:56Z

I think we can either drop Python3.10 (loses security support in a year; https://endoflife.date/python) or we can add a version check for the regridding tools

@Zeitsperre I do not have any preference here. Should these changes however be done in a different PR?

@cehbrecht What do you think?

I don't mind when we skip Python 3.10. If this regird PR will be "pending" one could already do the "skip" in another PR and merge it back in here again. What ever is easier for you :)

### What kind of change does this PR introduce?:  * Drops support for Python3.10 * Raises the minimum supported `numpy` to v1.26 ### Does this PR introduce a breaking change?:  Yes, the minimum Python is now 3.11 and the minimum `numpy` is at the absolute last version before `v2.0`. This is roughly following the [scipy SPEC-0 recommendations](https://scientific-python.org/specs/spec-0000/).

sol1105 · 2025-12-05T14:09:59Z

@Zeitsperre Do you have any objections against a simple merge commit without squash/rebase?

sol1105 added 3 commits August 13, 2025 16:22

Fixed regrid.ipynb and removed psymaps/psyplot dependency

2488d80

Removed psymaps / psyplot dependency for docs

42bd2df

sol1105 and others added 5 commits October 8, 2025 11:20

Fix for _grid_from_ds_adaptive for regional source grids

26a5f5c

Merge branch 'master' into regrid_updates_and_output_fixes

124b664

Merge branch 'master' into regrid_updates_and_output_fixes

c5affed

conda: temporarily installing xESMF from master branch

0293fba

conda: pin xarray for the moment

b396cab

Zeitsperre approved these changes Nov 24, 2025

View reviewed changes

sol1105 and others added 2 commits November 25, 2025 16:00

Added suggestions and fixes

549b041

[pre-commit.ci] auto fixes from pre-commit.com hooks

4b5398e

for more information, see https://pre-commit.ci

sol1105 added 2 commits November 28, 2025 20:37

Update HISTORY.rst

60d7c25

Added test for determine_grid_orientation; updated dependencies with …

854ae3c

…regards to xesmf

This was referenced Dec 1, 2025

Bump the actions group across 1 directory with 3 updates #468

Closed

Bump the python group across 1 directory with 4 updates #467

Closed

Zeitsperre added 4 commits December 2, 2025 11:35

drop Python 3.10

bfb4e30

update HISTORY.rst

13cba4c

update HISTORY.rst

825e4fe

pin numpy higher in environment.yml

d6ac67b

cehbrecht approved these changes Dec 5, 2025

View reviewed changes

Regrid updates and output fixes #447

Are you sure you want to change the base?

Regrid updates and output fixes #447

Uh oh!

Conversation

sol1105 commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist:

What kind of change does this PR introduce?:

Does this PR introduce a breaking change?:

Other information:

Uh oh!

coveralls commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 19869800153

Details

💛 - Coveralls

Uh oh!

sol1105 commented Sep 9, 2025

Uh oh!

Zeitsperre commented Nov 21, 2025

Uh oh!

sol1105 commented Nov 21, 2025

Uh oh!

Zeitsperre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Zeitsperre Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

sol1105 Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Zeitsperre Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Zeitsperre Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sol1105 commented Nov 25, 2025

Uh oh!

sol1105 commented Nov 28, 2025

Uh oh!

aulemahal commented Nov 28, 2025

Uh oh!

Zeitsperre commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sol1105 commented Dec 2, 2025

Uh oh!

cehbrecht commented Dec 2, 2025

Uh oh!

sol1105 commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sol1105 commented Aug 13, 2025 •

edited

Loading

coveralls commented Aug 13, 2025 •

edited

Loading

Zeitsperre commented Nov 28, 2025 •

edited

Loading