Skip to content

update parallel state following Megatron-LM parallel_state.py 2025.1#32

Open
1195343015 wants to merge 4 commits intoaliyun:masterfrom
1195343015:update-parallel-state
Open

update parallel state following Megatron-LM parallel_state.py 2025.1#32
1195343015 wants to merge 4 commits intoaliyun:masterfrom
1195343015:update-parallel-state

Conversation

@1195343015
Copy link
Contributor

  1. update parallel state following Megatron-LM parallel_state.py 2025.1
  2. add cp argument and cp_group for developing in future
  3. fix some small issues (like when world_size or comm_size is 1, there should be no communications)

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@1195343015
Copy link
Contributor Author

CC @zhouheyang-alibaba

@1195343015 1195343015 force-pushed the update-parallel-state branch from 1cdfeaf to eac4527 Compare May 30, 2025 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants