Skip to content

Add Energy Score metric for multivariate ensemble verification#720

Open
abhaygoudannavar wants to merge 5 commits intoNVIDIA:mainfrom
abhaygoudannavar:feat/energy-score-metric
Open

Add Energy Score metric for multivariate ensemble verification#720
abhaygoudannavar wants to merge 5 commits intoNVIDIA:mainfrom
abhaygoudannavar:feat/energy-score-metric

Conversation

@abhaygoudannavar
Copy link

@abhaygoudannavar abhaygoudannavar commented Feb 27, 2026

Description

Add energy_score class to earth2studio.statistics module implementing the multivariate generalization of CRPS (Continuous Ranked Probability Score) for ensemble forecast verification.

Closes #719

Why: Every major ensemble AI weather paper (GenCast, AIFS-ENS, FCN3, Atlas) reports Energy Scores, but Earth2Studio only has univariate CRPS. The Energy Score evaluates whether ensemble forecasts preserve spatial correlations across variables and grid points — critical for downstream tasks like tropical cyclone tracking.

Features:

  • Follows existing Metric protocol (ensemble_dimension, reduction_dimensions)
  • Supports configurable multivariate_dimensions for flexible Euclidean norm computation
  • Fair (unbiased) and standard estimator variants
  • Efficient pairwise distance computation via torch.cdist
  • Optional weighted reduction over additional dimensions
  • Pure PyTorch, no new dependencies

References: Gneiting & Raftery (2007), "Strictly Proper Scoring Rules, Prediction, and Estimation", JASA 102(477), 359-378

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.
  • The CHANGELOG.md is up to date with these changes.
  • An issue is linked to this pull request.
  • Assess and address Greptile feedback (AI code review bot for guidance; use discretion, addressing all feedback is not required).

Dependencies

None — pure PyTorch implementation.

Add energy_score class to earth2studio.statistics module implementing
the multivariate generalization of CRPS for ensemble forecast verification.

Features:
- Follows existing Metric protocol (ensemble_dimension, reduction_dimensions)
- Supports configurable multivariate_dimensions for flexible norm computation
- Fair (unbiased) and standard estimator variants
- Efficient pairwise computation via torch.cdist
- Optional weighted reduction over additional dimensions
- Pure PyTorch, no new dependencies

Includes comprehensive test suite (9 tests) with accuracy, non-negativity,
perfect ensemble, and multi-dimensional tests.

References: Gneiting and Raftery (2007), JASA 102(477), 359-378
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 27, 2026

Greptile Summary

Adds a well-implemented Energy Score metric for multivariate ensemble forecast verification. The implementation correctly follows the mathematical definition from Gneiting & Raftery (2007) and maintains consistency with the existing crps metric pattern.

Key strengths:

  • Mathematically correct implementation using efficient torch.cdist for pairwise distances
  • Supports both standard and fair (unbiased) estimators with appropriate validation
  • Flexible multivariate_dimensions parameter enables computing Euclidean norms across any combination of dimensions
  • Comprehensive test coverage including accuracy verification against reference implementation, edge cases, and mathematical properties (non-negativity, perfect ensemble)
  • Well-documented with clear docstrings explaining the formula and use cases
  • Follows the existing Metric protocol from earth2studio.statistics.base

Note: The PR checklist indicates CHANGELOG.md should be updated but no changes were made to it. Consider adding an entry to the "Added" section of version 0.13.0a0.

Confidence Score: 5/5

  • This PR is safe to merge with no code concerns
  • The implementation is mathematically sound, follows established patterns from the codebase (particularly crps.py), includes comprehensive test coverage (8 test functions covering functionality, accuracy, edge cases, and mathematical properties), and has appropriate error handling. The code is production-ready.
  • No files require special attention

Important Files Changed

Filename Overview
earth2studio/statistics/energy_score.py Implements Energy Score metric following existing patterns, with correct math, comprehensive error handling, and efficient PyTorch operations
test/statistics/test_energy_score.py Comprehensive test suite covering basic functionality, fair estimator, error conditions, accuracy verification, and mathematical properties
earth2studio/statistics/init.py Simple addition of energy_score import following existing module conventions

Last reviewed commit: 7e15172

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@abhaygoudannavar
Copy link
Author

@NickGeneva i wanted to know your opinion on this PR.

@NickGeneva NickGeneva added the 1 - On Deck To be worked on next label Mar 9, 2026
@NickGeneva
Copy link
Collaborator

Thanks for the PR @abhaygoudannavar will take a look!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1 - On Deck To be worked on next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🚀[FEA]: Add Energy Score metric for multivariate ensemble forecast verification

2 participants