Add DTW, nDTW, and SDTW trajectory metrics#12
Open
nalinraut wants to merge 1 commit intoAmeyaWagh:mainfrom
Open
Add DTW, nDTW, and SDTW trajectory metrics#12nalinraut wants to merge 1 commit intoAmeyaWagh:mainfrom
nalinraut wants to merge 1 commit intoAmeyaWagh:mainfrom
Conversation
Add Dynamic Time Warping based metrics for evaluating trajectories that may have different lengths or temporal alignment. These metrics are particularly useful for evaluating VLA models and policies using action chunking (e.g., ACT, Diffusion Policy). New metrics: - DTWDistance: Raw DTW distance using dynamic programming (lower=better) - NormalizedDTW: Mapped to [0,1] using exp(-DTW/(|R|*d)) (higher=better) - SuccessWeightedDTW: nDTW weighted by task success (SDTW = nDTW * Success) Key features: - Support for trajectories of different lengths (core advantage over MSE/ATE) - Tolerates temporal misalignment (hesitation, speed differences) - Optional custom normalization factor - Full torchmetrics.Metric compatibility with distributed training support - Comprehensive test suite and example usage Reference: Ilharco et al., "General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping," arXiv:1907.05446, NeurIPS ViGIL Workshop, 2019.
AmeyaWagh
reviewed
Feb 22, 2026
| from torchmetrics import Metric | ||
|
|
||
|
|
||
| def _compute_dtw(predicted: Tensor, reference: Tensor) -> Tensor: |
Owner
There was a problem hiding this comment.
Does this need to be an independenct function? can this be part of the metric class?
AmeyaWagh
reviewed
Feb 22, 2026
| accumulated[0, 0] = cost_matrix[0, 0] | ||
|
|
||
| # Initialize first column | ||
| for i in range(1, t_pred): |
Owner
There was a problem hiding this comment.
Can we avoid loops and use torch.linspace to index instead?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Dynamic Time Warping based metrics for evaluating trajectories that may have different lengths or temporal alignment. These metrics are particularly useful for evaluating VLA models and policies using action chunking (e.g., ACT, Diffusion Policy).
New metrics:
Key features:
Reference: Ilharco et al., "General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping," arXiv:1907.05446, NeurIPS ViGIL Workshop, 2019.