Add Energy Score metric for multivariate ensemble verification#720
Open
abhaygoudannavar wants to merge 5 commits intoNVIDIA:mainfrom
Open
Add Energy Score metric for multivariate ensemble verification#720abhaygoudannavar wants to merge 5 commits intoNVIDIA:mainfrom
abhaygoudannavar wants to merge 5 commits intoNVIDIA:mainfrom
Conversation
Add energy_score class to earth2studio.statistics module implementing the multivariate generalization of CRPS for ensemble forecast verification. Features: - Follows existing Metric protocol (ensemble_dimension, reduction_dimensions) - Supports configurable multivariate_dimensions for flexible norm computation - Fair (unbiased) and standard estimator variants - Efficient pairwise computation via torch.cdist - Optional weighted reduction over additional dimensions - Pure PyTorch, no new dependencies Includes comprehensive test suite (9 tests) with accuracy, non-negativity, perfect ensemble, and multi-dimensional tests. References: Gneiting and Raftery (2007), JASA 102(477), 359-378
Contributor
Greptile SummaryAdds a well-implemented Energy Score metric for multivariate ensemble forecast verification. The implementation correctly follows the mathematical definition from Gneiting & Raftery (2007) and maintains consistency with the existing Key strengths:
Note: The PR checklist indicates CHANGELOG.md should be updated but no changes were made to it. Consider adding an entry to the "Added" section of version 0.13.0a0.
|
| Filename | Overview |
|---|---|
| earth2studio/statistics/energy_score.py | Implements Energy Score metric following existing patterns, with correct math, comprehensive error handling, and efficient PyTorch operations |
| test/statistics/test_energy_score.py | Comprehensive test suite covering basic functionality, fair estimator, error conditions, accuracy verification, and mathematical properties |
| earth2studio/statistics/init.py | Simple addition of energy_score import following existing module conventions |
Last reviewed commit: 7e15172
Author
|
@NickGeneva i wanted to know your opinion on this PR. |
Collaborator
|
Thanks for the PR @abhaygoudannavar will take a look! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add energy_score class to
earth2studio.statisticsmodule implementing the multivariate generalization of CRPS (Continuous Ranked Probability Score) for ensemble forecast verification.Closes #719
Why: Every major ensemble AI weather paper (GenCast, AIFS-ENS, FCN3, Atlas) reports Energy Scores, but Earth2Studio only has univariate CRPS. The Energy Score evaluates whether ensemble forecasts preserve spatial correlations across variables and grid points — critical for downstream tasks like tropical cyclone tracking.
Features:
ensemble_dimension, reduction_dimensions)multivariate_dimensionsfor flexible Euclidean norm computationtorch.cdistReferences: Gneiting & Raftery (2007), "Strictly Proper Scoring Rules, Prediction, and Estimation", JASA 102(477), 359-378
Checklist
Dependencies
None — pure PyTorch implementation.