[Rllib] Modify StatelessCartPole APPO example to use new APPO attributes
#59894
Conversation
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the StatelessCartPole APPO example to address a timeout issue, likely by using new APPO configuration attributes. The changes involve enabling MeanStdFilter for observation normalization and adjusting training parameters by setting use_circular_buffer to False and increasing broadcast_interval. My main feedback is regarding a leftover TODO comment, which could cause confusion about the stability of MeanStdFilter.
| .env_runners( | ||
| env_to_module_connector=lambda env, spaces, device: MeanStdFilter(), | ||
| ) |
There was a problem hiding this comment.
You've enabled MeanStdFilter here, but the TODO comment on lines 25-26 mentions that it might cause NaNs during training. If this issue has been resolved, it would be great to remove the TODO comment to avoid confusion for future readers. If the issue is still present, perhaps a note explaining why it's being enabled despite the potential problem would be helpful.
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
simonsays1980
added
rllib-algorithms
2 participants
Description
The
StatelessCartPoleexample form APPO is timing out. This could be due to the latest changes in the APPO data pipeline. This PR modifies the setup of the example by using the new APPO attributes.Related issues
Fixes https://buildkite.com/ray-project/postmerge/builds/15188#019b8f6e-2850-465e-a98c-63c29fbf98f7/L4702
Additional information