New Use Case: Soil Moisture via SMOPS#3253
Merged
JohnHalleyGotway merged 8 commits intodevelopfrom Mar 30, 2026
Merged
Conversation
j-opatz
changed the title
Pull Request Test Coverage Report for Build 23659281761Details
💛 - Coveralls |
Closed
25 tasks
anewman89
requested changes
Mar 24, 2026
docs/use_cases/model_applications/land/GridStat_fcstGFS_obsSMOPS_soilMoisture_Python.py
Outdated
Show resolved
Hide resolved
docs/use_cases/model_applications/land/GridStat_fcstGFS_obsSMOPS_soilMoisture_Python.py
Outdated
Show resolved
Hide resolved
docs/use_cases/model_applications/land/GridStat_fcstGFS_obsSMOPS_soilMoisture_Python.py
Outdated
Show resolved
Hide resolved
docs/use_cases/model_applications/land/GridStat_fcstGFS_obsSMOPS_soilMoisture_Python.py
Outdated
Show resolved
Hide resolved
docs/use_cases/model_applications/land/GridStat_fcstGFS_obsSMOPS_soilMoisture_Python.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Andy Newman <anewman89@users.noreply.github.com>
anewman89
approved these changes
Mar 30, 2026
Collaborator
|
Note that I added @anewman89 to the METplus Developer team so that his future reviews will "count" as approving reviews. I'm proceeding with merging this PR based on Andy's previous review. |
metplus-bot
added a commit
that referenced
this pull request
Mar 31, 2026
j-opatz
added a commit
that referenced
this pull request
Mar 31, 2026
* add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow
* Update version 12.0.2 MET (#2896)
* use develop branch version of the component versions script to determine version numbers for creating docker images for releases
* updated configuration (#2901)
* fixed typo
* Feature 2891 develop existing builds modulefiles (#2909)
* Per #2891, updated for 6.1.0 release with coming soon messages added
* Per #2891, updating all modulefiles for 6.1.0
* Per #2891, fixing formatting
* Feature 2741 datsets and workflow (#2839)
* new branch testing
* marine and cryosphere datasets and workflow
* medium_range datasets and workflow
* pbl and precep files for datasets and workflow
* s2s_mjo and s2s files datasets and workflow
* Per #2741, updated METplus Workflow section
* first 5 short_range files
* Per #2741, rearranged and modified information for consistency.
* short_range directories starting with m or p
* short_range/UserScript files
* space_weather files
* tc_and_extra_tc and unstructured_grids files
* removing extra #
* fixes Julie suggested
* more fixes and new updates
* second pass of short_range files
* last directories clean up
* Tidy up UGRID use case doc file.
* Minor changes to tc_and_extra_tc docs.
* Minor changes to space_weather docs.
* Minor changes to short_range docs.
* Minor changes to s2s_mjo docs.
* Minor changes to s2s docs.
* Minor changes to precipitation docs.
* Minor changes to pbl docs.
* Minor changes to medium_range docs.
* Minor changes to marine_and_cryo docs.
* Clarifying on climo dataset.
* Adds INIT_END and INIT_INCREMENT to workflow.
* Removes unused conf item.
* Adds other timing control elements for completeness.
* Removes unused conf item.
* Removes unused conf item and switches min/max for readability.
* Adds additional timing control conf items for completeness.
* INIT_END is not set so changed to None.
* Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that.
* Adds other timing control for consistency.
* Adds other timing control for consistency.
* Adds other timing control for consistency.
* Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case.
* Clean up CyclonePlotter METplus Workflow.
* Adds clarity about the 60h lead time.
* Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools.
* Adds timing info for more clarity and consistency with template.
* Per #2741, correcting formatting
---------
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
* add run_metplus.py to bin directory upon install
* Revert "add run_metplus.py to bin directory upon install"
This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638.
* Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918)
* Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently
* change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263
* get older set of tar_files that includes jasper version needed to install gfdl tracker
* revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0
* remove args now that SSL cert for dtcenter site is renewed
* Per #2697, update version of conda environments to use envs created using python 3.12.0
* added scipy to python embedding environment because it is now a requirement of MET Python Embedding
* remove deprecated import
* update swpc env script
* add option to specify branch to use for METplotpy/METcalcpy/METdataio
* add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y
* simplify METplus Analysis branch override
* do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image
* fix warnings in python script
* fix error using numpy 2.X
* added multi-variate mode quick search keyword so use case is more discoverable
* updated config settings with new variables
* add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly
* revert back override of MET image
* changes to appease SonarQube
* removed duplicate entries
* rearrange args to appease SonarQube
* Updating METplus-6.1.0 Development Timeline
* Feature #2923 SonarQube for 6.1.0-beta2 (#2924)
* fix sonarqube findings
* update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults
* force https protocol to satisfy SQ
* copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method
* prevent future warning
* Feature 2744 nrl gfs goes (#2926)
* Use case config file from @briannen.
* Documentation template for this use case.
* Changes GRIBv2 record numbers to match subsetted forecast files.
* Updates to add new use case to automation.
* Updates input locations to the correct place where sample data are stored.
* Adds use case gallery image.
* Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation.
* Use case documentation file.
* Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes.
* Changes docker attempts to 20 instead of 10.
* Minor tweaks to docs from PR review.
* Feature #2697 python 3.12 round 2 (#2930)
* fix unit test to work with python 3.12 and 3.10
* per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions
* update recommended python version to 3.12.0
* per #2697, run unit tests using both py3.10 and py3.12
* add missing comma
* add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs)
* use scripts for python ersions to ensure 3.10 is not truncated to 3.1
* update version of python to use for RTD to 3.12
* added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated
* added instructions to create the metplus_dev environment so it can easily be created locally
* update required python for install
* update conda environments for use case tests to use versions that were created with python 3.12
* disable spacetime use case because pyngl is not supported in python 3.12
* change MJO use cases to use mp_analysis conda env instead of spacetime
* update python package versions
* install METplotpy and METcalcpy automatically if mp_analysis environment is used
* use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0)
* Feature 2745 nrl gfs asos (#2922)
* Add NRL GFS-ASOS use case
* Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy.
* Latest versions to use.
* Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast.
* Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file.
* Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height.
* Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups.
* Removes INIT_INCREMENT because only one INIT time is being processed.
* Turns off MPR output.
* Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations.
* Adds valid time to obs filename so it dynamically selects the correct date for the input file.
* Adds use case documentation gallery image.
* Renames gallery image filename and updates path in docs file.
* Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used.
* Fixes incorrect quotes in the use case documentation template for the gallery image path.
* Fixes incorrect quotes in sphinx gallery image file path.
* Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using.
* GHA/CI integration after adding sample data.
* Updates to input dirs.
* Correct input directory for obs.
* Avoids namespace conflict with standard conf item.
* Adjusts documentation for correct forecast leads.
* Removes MET_PYTHON_EXE from config.
* Removes verbosity settings from testing.
* Removes setting OBTYPE.
* Adds more comments to interpolation function.
* Adds many comments about read_asos_ceil.py.
* Adds info about the third Python embedding script gfs_025_interp_funcs.py.
---------
Co-authored-by: Tracy Hertneky <hertneky@ucar.edu>
Co-authored-by: j-opatz <jopatz@ucar.edu>
* fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen
* update instructions to download cartopy static files
* Feature #2863 initial conda recipe (#2900)
* initial implementation of conda recipe to install MET executables and METplus wrappers
* add setup tools to fix incorrect METplus wrappers on install
* prevent conflict when isatty is defined in multiple places
* update version of MET to 12.0.1 to include pb2nc bugfix
* remove specific versions of compilers that are no longer needed
* use printf to properly add line break
* turn off fortran optimization to prevent issues with NHC-provided code
* remove setting z lib flag because it shouldn't be necessary to compile MET
* remove fortran flags that are not needed
* skip if attempting to build on windows
* explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed
* update website URL and change maintainer to GitHub username
* prep for bugfix release
* update MET v12.0.2 hashes
* remove unused env vars and sym link of ar
* add back env vars that are needed, clean up
* set env var to easily access the parm directory for use case files
* specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe
* added more about info and changed maintainers item to correct name
* update license info
* add tests to ensure MET executables and required python imports are available
* clean up comments so they can be easily removed when submitting to conda-forge
* formatted changes based on conda-forge linter
* add cmake because it was missing from conda-forge build
* specify different dynamic library name for netcdf C++ to match what is available for osx-64
* change specific gcc compiler version line
* revert change to try to handle differing netcdf c++4 dynamic library names
* change wget to curl because wget is not available in conda-forge env
* formatting for conda forge linter
* skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10
* add license files for library dependencies
* specify linux compiler versions
* fixed typo in fortran version spec
* add make and pip to build reqs
* fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows
* add install of run_metplus.py in path
* clean up comments to be more accurate
* changed format of exact version requirements
* add instructions to build/test a conda recipe
* set parm base based on python version
* specify python version and numpy requirements
* changes suggested by @JohnHalleyGotway in the pull request review
* made formatting fixes suggested by @jprestop in the PR review
* per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3
* update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml
* updates based on feedback from conda-forge PR
* get eckit/atlas from metview on conda-forge instead of installing it
* change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure
* add description
* Feature 2743 nrl gfs cloud analysis (#2850)
* Initial commit of Use Case
* Adding use case image
* Updating documentation paths
* Updating processing
* Fixed typo
* Updated documentation typos
* Trying to get use case tests to run
* Fixed typo in use case name
* Turned on MODE run for testing
* More testing
* Updating data path
* Trying updated docker settings
* Updating text in .conf file
* Testing time for all mode runs
* Moving use case to new area
* Adding masking to the use case
* Fixing mask location
* Adding CI overrides
* Updating process list
* Fixing Wrong lead times
* Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py
Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
* Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py
Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
* Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py
Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
* updating to be clouds #8
* Update use_case_groups.json
* Updating to be clouds #8
* Some refactoring of the UserScript section.
* Missed an indent.
* Removing line with space confusing the RST formatter.
* Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST.
* Fixed error in use_case_groups.json
* Updates to fix one sonar cube error
* Changing use case to false
* Minor changes to UserScript wording.
* Addresses logger variable formatting issue.
---------
Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu>
Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
* updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions
* Feature 2771 subprojects (#2942)
* Per #2771, attempting to add MET User's Guide to ToC
* Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide
* Per #2771, adding COMPONENTS section
* Per #2771, playing with menu options
* Per #2771, playing with menu options
* Per #2771, try to add scroll bar functionality to ToC
* Per #2771, attempt two to add scroll bar
* Per #2771, attempt three add scroll bar for ToC
* Per #2771, attempt four to get scroll bar for ToC
* Per #2771, attempt to make logo area sticky
* Per #2771, attempt to fix problems with sticky logo
* Per #2771, attempting to fix overlap of sticky logo over ToC
* Per #2771, attempting to fix overlap of sticky logo over ToC
* Per #2771, attempting to fix overlap of sticky logo over ToC
* Per #2771, attempting to fix overlap of sticky logo over ToC
* Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header
* Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header
* Per #2771, menu updates
* Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu)
* Per #2771, replace reference to feature branch with latest
* Feature 2937 update data physics tendency (#2938)
* Update config file to match updated data
* redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked.
This may cause issues when running the use case in a CI environment
* Update config file to match updated data
* Replaced plot with new plot using updated data.
* Updated plot using latest data
* Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked.
* updates to conf files for the vertical profile plotting
* Updated thumbnails for new physics tendency plots generated from new data
* updates made to reflect changes due to new data
* Modified instructions on location of output file
* remove the redirection command, it does not work
* Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py
removed text that was erroneously copied
* pass output directory to script to write output in correct location
* remove unused imports
---------
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946)
* sq fix
* move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints
* handle SQ complaints for duplicate code and code smells
* fix paths to new locations of common scripts in documentation
* refactor to resolve SonarQube complaints
* more SQ - specify exception and start to reduce cognitive complexity
* clean up SQ complaints
* remove a few more code smells
* more code smell cleanup
* Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945)
* update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails
* fix imagemagick install by updating apt packages
* Feature #2941 v6.1.0-beta2 (#2950)
* add release notes
* update version for beta2 release
* update beta2 release date
* update info about DockerHub images for beta2
* update date of release
* Update docs/Users_Guide/release-notes.rst
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
* per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions
* disable doxygen by default
* fix formatting of note
---------
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
* update version for dev towards rc1
* Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold.
* Feature 2953 installation files (#2958)
* Per #2953, update Orion file for Python 3.12
* Per #2953, updating Existing Builds page for installation on Orion
* Per #2953, updating Casper file for Python 3.12
* Per #2953, adding script for METplus-6.1 using Python 3.12
* Per #2953, updated installation information for Casper
* Per #2953, updated compiler for Orion
* Per #2953 updating compiler in Existing Builds for Orion
* Per #2953, added qhull needed for atlas on hera
* Per #2953, remove qhull as it was actually causing problems on hera
* Per #2953, updating hera file for Python 3.12
* Per #2953, updating with hera information
* Per #2953, updating jet file for Python 3.12
* Per
* Per #2953, adding internal/scripts/installation to paths-ignore
* Feature #2560 Update Fire use case (#2956)
* add quotation marks around file path if it includes spaces
* update use case to read WRF fire field on subgrid directly instead of using Python Embedding
* update instructions for updating web server data to use MET scripts
* add a test for python embedding input
* fix issue introduced where double quotes are added
* prevent incorrect error when running PyEmbedIngest
* fix URL link
* update use case doc to note that Python Embedding is no longer used
* remove keyword to set up Python Embedding in automated tests because it is no longer used
* per PR review suggestion from Copilot, add comment to describe why variable is set
* override output units/level
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* fix formatting of set_attr options
---------
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* added suggestions from conda-forge code review
* force using python 3.10 locally for now
* fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base
* remove dependencies that don't need to be listed because they are made available by other dependencies
* add MET license file
* remove GS Fonts due to licensing issues and it is optional
* prevent warning for numpy because it is actually used
* add back libopenblas for linux only
* Feature 2746 sfs use cases (#2954)
* Adding Soil Moisture averaging case
* Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files.
* Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5
* Adding a thumbnail for the use case
* Adding updated soil moisture use case
* Updating title line
* Adding Use Case image
* Updating docs
* Trying to fix use case image
* Updating documentation
* Updating documentation
* Fixing documenation
* Updating documentation
* Updating directory
* Adding use case data
* Adding the s2s_soil_moisture directory
* Updating documentation location
* Adding Documentation README
* Adding config override for actions
* Fixing typo
* Testing Config overrides
* Testing new timeframe
* Changing CI overrides
* Renamed files and moved them to s2s_soil_moisture directory
* Updated file to change ERA to ERA5Land
* Moved file to the correct directory and updated
* Changed file name
* Fixed directory name
* Fixed directory name
* Fixed directory name
* Added test for the soil moisture use case
* Adding another use case
* Setting tests to false for pull requests
* Fixed image file name
* Testing sonar cube change
* Turning a test back on to check a change
* Update sfs_gsl_model_wrapper.py
* Testing a sonar cube update
* Updated documentation as per new template
* Updated documentation as per new template
* Gallery path update
* Update use_case_groups.json
* Turning tests off for PR
* Removed boilerplate language
* Changing log level
---------
Co-authored-by: Mrinal Biswas <biswas@ucar.edu>
Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com>
* add example of using the allow_empty extra arg keyword
* change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions
* fix tests that check python versions
* update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant)
* fix incorrect label
* Feature 2904 tif fuse case (#2987)
* first pass over doc, basic files. Needs config update and finalize doc
* add rioxarray conda environment
* update out-of-date info
* added image, updated docs, attempt to resolve compile issue
* removing space for error
* updated remaining files, adjusted docs
* fix incorrect naming for another use case that had errors in use case doc rendering
* fix typos and formatting
* fix path to thumbnail image
* add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env
* fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided
---------
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* Hotfix to Release Guide instructions for MET to fix a minor formatting issue.
* change version of sphinx-rtd-theme to a version that is available on conda-forge
* Feature #2962 RMWAnalysis wrapper (#2985)
* per #2962, begin implementation of RMWAnalysis wrapper
* start basic use case conf
* read BOTH_VAR<n> variables into data field
* change pycharm python interpreter
* ignore file generated by building docs locally
* update use case to read input data
* Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper
* per #2962, added thumbnail for use case documentation
* fix incorrect formatting
* per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string
* Bugfix #2890 develop - prevent incorrect error for skipped time (#2988)
* per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed
* add docs for setting runtime freq for Example wrapper
* prevent crash checking an init or valid time that is a wildcard
* clean up function docs
* Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose
* update metplus_dev environment to include the documentation requirements
* Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set
* Feature #2984 v6.1.0-rc1 (#2991)
* add release notes for rc1
* update DockerHub rc1 releases
* fix indentation
* Per #2984, updating title of issue 2863
---------
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
* change deprecated maintainer to label
* prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well
* remove docker image after pushing to DockerHub to free up disk space
* add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed
* remove problematic run-name logic because PR merge that will trigger the run has the information we need
* remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml
* rename job so workflow looks more similar to testing
* rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub
* add release note for missing bugfix issue
* push fix when no latest run for main_v6.1 branch exists
* do not run anything in testing workflow for -ref branches
* update version for development towards 6.2.0-rc1
* Feature 2932 use case amdar pbl (#2989)
* Enhanced AMDAR PBLH METplus use case
* Fixes title underline length error.
* Adds missing .csv for airport filename in the conf file.
* Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE.
* Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails.
---------
Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
* Feature #2998 develop - updating truth data instructions (#3001)
* per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info
* add note to delete the temporary branch generated for updating develop-ref
* enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS
* clean up tests
* Feature 2953 installation files (#3004)
* Per #2953, update Orion file for Python 3.12
* Per #2953, updating Existing Builds page for installation on Orion
* Per #2953, updating Casper file for Python 3.12
* Per #2953, adding script for METplus-6.1 using Python 3.12
* Per #2953, updated installation information for Casper
* Per #2953, updated compiler for Orion
* Per #2953 updating compiler in Existing Builds for Orion
* Per #2953, added qhull needed for atlas on hera
* Per #2953, remove qhull as it was actually causing problems on hera
* Per #2953, updating hera file for Python 3.12
* Per #2953, updating with hera information
* Per #2953, updating jet file for Python 3.12
* Per
* Per #2953, adding internal/scripts/installation to paths-ignore
* Per #2953, updating existing builds page.
* Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005)
* per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums
* Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated
* Feature #3008 Support lead months/years in template substitution (#3009)
* clean up code formatting
* Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now
* reduce SonarQube findings by reducing cognitive complexity. Clean up documentation
* Feature 2857 support (#3012)
* Per #2857, updated User Support section and also modified case of other headers
* Per #2857, resolved duplicate explicit target name warning.
* Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution.
* add swpc_metpy env to GHA workflow to create conda envs
* pull changes from metplus-feedstock
* reduce required python version to match metplus/PYTHON_VERSION_MIN
* add met_tool_wrapper use case to existing group
* Feature 2999 improve diffs for met tests (#3019)
* added some temp code to check for .log files being diffed
* removed checks for .log files / reverted
* added some more detail to output, including number files
* changed calc_transport.log to calc_transport.out and updated ref in docs
* skip .log files
* changes to diff_text_lines
* added temp check for files without extensions
* added temp check for files without extensions / fixed
* don't compare files without extensions, e.g. file_list in MET tests
* forgot to hit save after deleting a couple lines before the last commit...
* improved handling of nc files
* enhanced numeric diffs for nc files
* implemented logic for setting tiny values to zero as pper MET feature 3134
* implemented diff checking on sig figs
* slight tweak to order of equality checks
* added debug line
* more debugging
* fixed _round_sig_figs
* fixed bug in _set_zero
* another fix
* reinstated rounding precision for diffs after trying only sig fig rouding
* bumped SIG_FIG for diffs to 7
* fix to _round_sig_figs
* simplified _is_number()
* some tidying
* refactor/cleanup to remove code smells reported by SonarQube
* more cleanup
* added unit tests for skip file extensions logic and new equal comparison functions
* add comments to describe each list of file extensions
---------
Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu>
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027)
* Feature 2921 develop sphinx macports (#3025)
* Per #2921, adding documentation for setting up a sphinx environment using MacPorts
* Per #2921, adding context to explain why a developer may need to run these commands.
* Feature #2575 TCI use case - generic model (#2768)
* Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI.
* per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models
* cleanup
* merged (old) develop and resolved conflicts
* fixed incorrect merge
* update use case doc file to more closely match use case documentation template
* replace bad quotation mark character and fix indentation
* fix formatting
* more formatting changes
* fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts
* Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data.
---------
Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
* Feature #2980 refactor PCPCombine file search (#3017)
* per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper
* fix typo in name of temporary var list
* clean up logic for SonarQube
* handle if temp var (field) list is None or an empty list
* fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files
* refactored logic into function to reduce duplicate code
* move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run
* update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error
* add unit test to capture bug where ExtractTiles is failing
* fix bug from previous commit by settings output template/dir from FCST/OBS
* update test to run just FCST and just OBS instead of always both
* change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case
* clean up formatting complaints
* reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent
* fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set
* reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds
* Only require input template to be set for run methods other than USER_DEFINED and SUM.
* add unit test for USER_DEFINED run method
* refactor to reduce duplicate code
* add pytest fixtures to reduce duplicate code in unit tests
* improve/expand SUPPRESS_WARNINGS logic
* add argument to skip logging when input files cannot be found and just return True/False
* clean up formatting
* refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed
* update unit tests to match changes to wrapper
* fix ExtractTiles to not report errors when files are not found (previously reported warnings)
* reduce cognitive complexity
* split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes
* refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly
* clean up formatting
* clean up formatting
* Feature 3014 sfs updates (#3020)
* Adding graphic
* Updating python embedding script location
* Updating environment
* Updating environment
* Removing bad path
* Adding CI override
* Updating documentation
* Updating documenttion again
* Take 3
* Another documentation update
* Updating use case
* Updating the order of processes
* Updating documentation
* Cleaning up unused variables
* Updating documentation
* Adding climatology. Testing will fail b/c haven't uploaded data
* Adding Series Analysis climatology and updating documentation
* Another doc update
* Fixing valid_beg
* fixing time
* Fixing documentation typo
* Fixed another typo
* Fixing log verbosity and block size
* Updating some code
* Fixing typo
* Testing removal of file_type
* Removing file type
* Setting tests to false for pull requests
* minor grammar, spelling updates
* Fixing increment
* Fixing Log level
* Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf
* editing plot title
---------
Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu>
Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com>
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* Feature 2723 use case sep climates (#3030)
* added config and docs. Added use case to testing files. Still needs img, testing
* updated config with file_type, use case img
* Small grammar change
---------
Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu>
* Feature 2390 ctp hum index (#2981)
* Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs).
* Cleanup of extra debugging and development blocks.
* Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs).
* Minor changes to print statements.
* Major cleanup.
* Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy.
* Conf file for this use case.
* Python embedding script for FCST for CTP.
* Changes for UserScript and GenVxMask.
* Changes for function signature to take a station_id argument.
* Python embedding script to compute Humidity Index from UFS/HR1 forecast data.
* Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file.
* Updates to account for new station_id argument by calc_ctp function in METcalcpy.
* Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data.
* Removes erroneous code from script that this script was copied from.
* Removes unused code from development and print statements from development.
* Removes print statements used during development.
* Removes unused code from development and print statements from development.
* Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options.
* Turns down verbosity for PointStat.
* Adds print statement that the Python embedding script is starting for log purposes.
* Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy.
* Adds debug message for printing the station being processed, with the default turned off.
* Major updates to documentation file for use case.
* Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask.
* Cleanup of use case config gile to remove unused items.
* Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min.
* Removes unused functions from forecast Python embedding scripts.
* Switches to F strings and wraps some print statements in DEBUG flag.
* Switches to F strings and adds script name to print statements.
* Adds space around print statements to aid log file inspection.
* Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file.
* Adds controlling DEBUG level for observation Python embedding scripts.
* Updates documentation URL in config file.
* Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA.
* Updates for sample data and running the use case in GHA.
* Typo in the JSON file.
* Changes the MET_PYTHON_DIR to the installed location rather than the source code location.
* Adds user script to create graphic from PointStat MPR output.
* Adds use case gallery image and updates observation dataset info in documentation.
* Fixes path to config file include in documentation.
* Adds new user script for plotting to the documentation file.
* Updates list of expected output files to include UserScript outpout and PB2NC output.
* Changes to eliminate code smells.
* Finalizes forecast dataset description and scientific objective.
* Adds info on additional Python packages required for Python embedding and UserScripts.
* Removes extra #.
* Removes uneccesary imports in user script.
* Adds more clarity that METcalcpy is a required Python package for Python embedding.
* Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts.
* Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3.
* Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts).
* prevent exception when trying to compare a masked (nan) value to a number
* turn off use case
* added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images
* added catch for log10(0) (#3043)
Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu>
* Feature #2577 develop - WRF Hurricane Matthew use case (#3036)
* Feature #2577 WRF Hurricane Matthew use case (#3015)
* per #2577, add use case files
* turn off use case that shouldn't be on
* fix path to data and scripts
* fix path to plotting scripts
* added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy
* fix name of wrf python package
* override input and output directories in WRF plotting script
* clean up script re: SonarQube
* renaming variables for SonarQube
* turn off 100-m wind speed plot because zlev files are not included in the input data
* added missing closing single quote
* add version of package that was obtained for reproducibility
* refactor WRF plotting script to split logic into functions to reduce cognitive complexity
* try creating output directory before running reformatter
* METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory
* change approach to reference plot directory
* add WRF and MADIS file types to quick search keywords and start doc for new use case
* fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user)
* write log file to logs directory and rename it to include .log extension
* add more information for use case doc
* turn off use case for PR
* refactor map functions script to reduce cognitive complexity for SonarQube
* update scientific objective with input from @jaredalee
* update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now
* test that changes to use case did not break anything
* Feature 2533 ismn use case (#3051)
* Initial config file for this use case.
* Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT.
* Latest param file changes.
* Documentation file for this use case.
* Updates documentation URL in config file.
* Updated conf file that produces matched pairs for soil moisture from PointStat.
* Updates to documentation for ISMN use case.
* Adds UserScripts for plotting.
* Latest conf file settings for use case.
* Use case gallery image.
* Adds summary using NEAREST for point_stat matched pairs.
* Adds path to gallery image for use case.
* Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change.
* Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required.
* Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases.
* Minor change to use case documentation.
* Initial work for adding ISMN to Verification Datasets guide.
* Adds documentation for the ISMN dataset in the Verification Dataset Guide.
* Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide.
* Adds new use case to testing suite after adding sample data.
* Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion.
* Forgot a space.
* Update to MET 12.0.3 bugfix version (#3059)
* Update to MET 12.0.3 bugfix version.
* Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN".
* Feature #3054 develop scan_for_CVEs (#3055)
* Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub.
* Per #3054, update the Release Guide with instructions about updating the GHA release workflows.
* Per #3054, fix workflow typo.
* Per #3054, update release guide formatting.
* Per #3054, add CMD_LOGFILE logic to the time_command function.
* Per #3054, refine release guide instructions.
* Per #3054, refine release guide instructions.
* Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0.
* Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions.
* Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment.
* Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed.
* Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060)
* Replace metviewer 6.0.0 with 6.0.1 (#3065)
* updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases
* rotate authorship as was done in main_v6.1 branch
* fix typo
* update version of METexpress for 6.1 Coordinated Release
* add look up for upcoming 6.2 coordinated release
* Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074)
* update instructions to create a new project board to use templates (#3075)
* update instructions to create a new project board to use templates
* improve instructions for updating Zenodo
* Apply suggestions from code review
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
---------
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* develop docs v6.2.0 release timeline (#3083)
* Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET.
* Removing accidental new file.
* Feature #3068 DataIngest wrapper (#3084)
* clean up
* per #3068, add utility function to download files and added unit tests to confirm it works as expected
* return from function is uncompress extension is not present
* remove unnecessary close() calls because with handles closing of file handles
* Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes)
* per #3068, create basic components of DataIngest wrapper
* Add argument to getraw function to skip replacement of double slash with single slash. The replacement was added to prevent errors with paths that contain double slashes in MET config files, but this causes issues with URLs.
* clean up to remove unnecessary variable
* Rename DATA_INGEST_<n>_URL/LOCAL_PATH to INPUT/OUTPUT_TEMPLATE to be consistent with other wrappers. Fix incorrect has_option config function name. Improve logging so it is clear that a file is currently being downloaded. Added skip-if-output-exists logic
* added basic use case for new wrapper
* Handle gz extension as gzip file. Refactor new functions to reduce cognitive complexity
* add option to auto decompress files before writing to disk
* add use case documentation
* add new use case to run in automated test suite
* added config options for INPUT/OUTPUT_DIR to be consistent with other wrappers -- added workaround for getdir function to prevent replacement of double slash in URLs
* fixed typo
* added documentation for config variables for new wrapper
* turn off new use case before creating PR
* remove test that takes a very long time to download file
* removed deprecated config variable
* fixed bug in enhancement to config getdir to properly handle skip of double slash handling
* improve logging for readability
* change lead sequence in example because it is not used
* Per #3068, add unit tests for new DataIngest wrapper
* fix typo
* refactor automated test scripts to reduce duplication in logic to run commands
* improve logging so it is more clear that a diff test failed because it is likely that the truth data does not exist yet (e.g. for new use cases) and exit 2 to match return code for a use case that contains actual diffs in the output
* Update the METplus-6.1 version of MET from 12.1.0 to 12.1.1 after the first bugfix release. (#3093)
* Per #3087, add support for setting message_type_group_map in PB2NC and EnsembleStat wrappers (#3092)
* update climo stdev variable name after input data was updated by @j-opatz
* change sonar host url to env to view url for quality gate scan results
* Fix FSTDEV typo in statistics_list.rst
* Update release-notes.rst for the METplus-6.2 Release Schedule
* Feature #3028 RMWAnalysis configs (#3094)
* Per #3028, added support for many new rmw_analysis MET config options. Refactor logic to read METplus configs and handle MET config env vars to loop over names and handle them consistently. Reordered variables to match new order in RMWAnalysis_default config file
* fix missing line break that caused warnings in doc build
* change input URLs to obtain files from DTC web server because MADIS data changed since the truth data was created
---------
Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu>
* Feature #3069 Generate run times using templates (#3089)
* removed unused imports
* removed unused import
* Remove loop_over_times_and_call function because time looping should be handled in RuntimeFreq wrapper. Report an error if a wrapper does not properly inherit from RuntimeFreq or implement its own version of run_all_times
* update unit tests because CommandBuilder run_all_times now reports an error
* added a fixture to create empty files for testing file I/O logic in the unit tests
* cleanup to reduce cognitive complexity, remove unnecessary variable, and clean up whitespace for readability
* Per #3069, create get_files_and_time_info function to reuse logic to find files using a directory and template for file window logic and new logic to generate list of times to run based on existing files on disk. Add logic to time_generator to support TIME_GENERATOR_INPUT_DIR/TEMPLATE to generate init or valid times to process. Add unit tests to ensure that correct, sorted list of init/valid times are generated. Add helper function for unit tests to ensure that the expected keys for each time dictionary are present and call helper function from tests that use time_generator to generate run times from beg/end/increment and time list. Added stub to implement similar logic to extract available forecast leads for each init/valid time
* Per #3069, use TIME_GENERATOR_INPUT_DIR/TEMPLATE to get list of forecast leads that correspond to the current init or valid time being processed. Updated tests to include this functionality
* clean up docstring comments
* Per #3069, add tests to ensure that logic to allow multiple time generator dir/templates and get the intersection of the run times works as expected before implementing it
* Per #3069, enhance time generator dir/template logic to support multiple templates and get the intersection of the run times
* add test for a single directory with multiple templates and add helper functions to reduce duplication
* whitespace
* add function to intelligently split dir/template values so that the dir will contain the path without any template tags and the template will contain any filename template tags so users can include filename template tags in _INPUT_DIR variables and file window and time generator parsing will still work as expected. Added ample tests to ensure correct behavior
* correct comments
* update test to include full list of existing wrappers
* clean up tests and add test for case that could occur in containers and breaks dir/template split logic
* implement fix that properly splits dir/template when a template is used in the root (/) directory
* check if either TIME_GENERATOR_INPUT_DIR or _TEMPLATE are set to execute logic to get runtimes based on file templates since logic to properly split the path into dir/template prevents the requirement of using both config variables
* added helper function to reduce cognitive complexity and make code more readable
* reduce duplication by creating variable to store test data templates that are referenced in multiple tests
* Per #3069, add support for time generator templates that only include valid time (obs data) and properly filter init/lead times accordingly.
* Per #3069, add documentation to describe how to use new functionality with examples
* Per #3069, improve logging to be more clear which times were found from templates and when no forecast leads were found for a given init/valid time
* Added more debug logging regarding files/times found from searching. Removed function that was only called once
* update use case docs to use latest template content
* per #3069, add new use case to demonstrate the use of the new functionality to generate run times based on file templates
* allow Example wrapper to support multiple inputs, because why not
* added quick search keyword for times from template functionality
* update use case docs to match template
* Filter analysis files in first pass only if looping by valid time. Clean up comments to be more clear/accurate
* create helper function to reduce cognitive complexity
* Per #3069, enhance template-based time generation to support {instance} and {custom} template tags.
* reduce duplication
* fix formatting that breaks in py 3.10
* Log number of times each command was run from each wrapper. Rename function to match its new purpose
* refactor to reduce cognitive complexity
* remove trailing semicolon from executable name in log
* per PR feedback from @CPKalb, add missing METplus config variables that are ignored when using time generator input template
* include instance name when logging report of successful/failed runs
* group new use cases
* hotfix for #3069 -- add support for using now/today template tags in the time generator template and allow TIME_GENERATOR_INPUT_DIR to be unset in favor of setting the entire template in TIME_GENERATOR_INPUT_TEMPLATE
* SonarQube clean up for v6.2 (#3108)
* remove doxygen files because they aren't used
* remove commented code
* fix formatting of use case docs that get flagged as SQ issues incorrectly
* SQ: fix issues with Dockerfiles
* SQ: replace unused vars with underscore
* SQ: reduce cognitive complexity
* SQ: rename CommandBuilder variable isOK to is_ok
* SQ: various issues resolved
* fix incorrect variable used in read_filters function
* use env var value instead of hard-coded extension
* SQ: various clean up to use case scripts to resolve SonarQube issues
* fix format variable used for time series
* remove check for success of calls to add_met_config_dict_list because it is not necessary
* return False instead of None to be consistent with the other return values in the function
* remove test that is no longer needed after removing status check
* Feature 3041 appendix a release notes (#3101)
* Per #3041, updating appendix A
* Per #3041, updating release notes to reflect need to update Appendix A
* update link to MET Python Embedding reqs to use latest instead of feature branch
* Update docs/Release_Guide/release_steps/update_appendix_a_python.rst
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* Update docs/Release_Guide/release_steps/update_appendix_a_python.rst
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
---------
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* Feature #3069 part 2 - requests instead of urllib (#3117)
* update unit tests to download from another URL that is more reliable
* Per #3069, refactor to use requests packages instead of urllib to increase download speed
* add scripts to create new conda docker env that contains the requests package needed to run the DataIngest wrapper
* install requests package for unit tests
* error in init step if requests package is not installed to prevent failure when running DataIngest wrapper
* add requests to workflow used to create conda envs in docker
* added version number for package
* use requests env for DataIngest use case
* replace mix of spaces and tabs with just spaces
* change long line to multiple
* document info about requirements for python package
* install requests package for sonarqube GHA workflow that runs unit tests
* Feature #3033 ascii2nc -inputrx (#3114)
* remove doxygen files because they aren't used
* remove commented code
* fix formatting of use case docs that get flagged as SQ issues incorrectly
* SQ: fix issues with Dockerfiles
* SQ: replace unused vars with underscore
* SQ: reduce cognitive complexity
* SQ: rename CommandBuilder variable isOK to is_ok
* SQ: various issues resolved
* fix incorrect variable used in read_filters function
* use env var value instead of hard-coded extension
* SQ: various clean up to use case scripts to resolve SonarQube issues
* fix format variable used for time series
* remove check for success of calls to add_met_config_dict_list because it is not necessary
* return False instead of None to be consistent with the other return values in the function
* remove test that is no longer needed after removing status check
* Per #3033, allow ascii2nc wrapper to write a file list file to handle use case that has a large number of input files
* remove LOG_LEVEL that is not typically set in use case conf files
* per #3033, add documentation for new config variable
* add quotes around -inputrx argument
* Per #3033, allow ascii2nc input template to be a directory to be searched. Do not write file list file if any of the inputs are a directory or python script. Do not create file list files for other ReformatPoint wrappers pb2nc, madis2nc, and point2grid
* add new config to basic use case
* Per #3033, update ISMN use case to use -inputrx and directories. Also clean up PROCESS_LIST to make it more readable
* Per #3033, add unit tests to ensure command line arguments are set properly
* fix syntax error using python 3.10
* Feature #3110 orography (#3113)
* remove doxygen files because they aren't used
* remove commented code
* fix formatting of use case docs that get flagged as SQ issues incorrectly
* SQ: fix issues with Dockerfiles
* SQ: replace unused vars with underscore
* SQ: reduce cognitive complexity
* SQ: rename CommandBuilder variable isOK to is_ok
* SQ: various issues resolved
* fix incorrect variable used in read_filters function
* use env var value instead of hard-coded extension
* SQ: various clean up to use case scripts to resolve SonarQube issues
* fix format variable used for time series
* remove check for success of calls to add_met_config_dict_list because it is not necessary
* return False instead of None to be consistent with the other return values in the function
* remove test that is no longer needed after removing status check
* Per #3110, add support for land_mask and topo_mask dictionaries in EnsembleStat wrapper
* Per #3110, add support for lapse_rate_correction and msl_agl_conversion dictionaries for PointStat and EnsembleStat
* clean up method doc
* change check/error to an assert because it is meant to prevent a developer from calling the method with an incorrect argument
* remove redundant call to parse_var_list and clean up formatting
* add app name to call to parse_var_list so that FCST/OBS_STAT_ANALYSIS_VAR<n> variables are also supported
* update test to reflect change in log message
* Per #3310, add structure to support setting variables at a per-field basis instead of relying on *_VAR<n>_OPTIONS that require correct MET formatting. This structure provides the groundwork needed to complete issue #2357
* clean up formatting of field options
* remove unused import
* correct method doc
* refactor to reduce cognitive complexity
* Per #3110, add support for setting per field land_mask.flag, topo_mask.flag, topo_mask.use_obs_threhs, topo_mask.interp_fcst_thresh, lapse_rate_correction.apply_to, lapse_rate_correction.value, msl_agl_conversion.apply_to, msl_agl_conversion.apply_from, msl_agl_conversion.thresh, and msl_agl_conversion.msl_to_agl for both PointStat and EnsembleStat wrappers
* add support for APP_NAME_FCST/OBS_* variables, e.g. GRID_STAT_FCST_INPUT_TEMPLATE, to start moving away from FCST/OBS_APP_NAME variables, e.g. FCST_GRID_STAT_INPUT_TEMPLATE -- The latter will still be supported and a warning will be output if both formats are used in a configuration
* fix conflict resolution
* Per #3110, added documentation for new config variables to set new configs per obs.field
* change . to _ for METplus config names in helpe…
metplus-bot
added a commit
that referenced
this pull request
Mar 31, 2026
georgemccabe
pushed a commit
that referenced
this pull request
Mar 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Testing
Describe testing already performed for these changes:
Use case was run successfully on Seneca.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
Check documentation out for any errors. Look for any Github automated testing errors associated with this use case (other issues should be reported, but are not as big a concern).
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Do these changes include sufficient testing updates? [Yes]
Will this PR result in changes to the test suite? [Yes]
If yes, describe the new output and/or changes to the existing output:
Use case will add new output
Do these changes introduce new SonarQube findings? [No]
If yes, please describe:
Please complete this pull request review by [Fill in date].
3/31/2026
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: Coordinated METplus-X.Y Support project for bugfix releases or METplus-Wrappers-X.Y.Z Development project for official releases