Skip to content

[Sonic MoE] Support Sonic MoE Router#297

Draft
hushenwei2000 wants to merge 1 commit intoPaddlePaddle:developfrom
hushenwei2000:sonic_moe
Draft

[Sonic MoE] Support Sonic MoE Router#297
hushenwei2000 wants to merge 1 commit intoPaddlePaddle:developfrom
hushenwei2000:sonic_moe

Conversation

@hushenwei2000
Copy link
Collaborator

需要拉取 hushenwei2000/sonic-moe@5e80341 更改了 Sonic MoE 源码的:

  1. 先进行激活函数(softmax/sigmoid),再进行 TopK
  2. 支持 sigmoid 激活函数

目前仅测试了 GLM4.5 标准测试。测试结果

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant