-
Notifications
You must be signed in to change notification settings - Fork 618
add mtp develop guide doc #3956
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: zouyida2052 <zouyida2002@gmail.com>
Signed-off-by: zouyida2052 <zouyida2002@gmail.com>
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request adds a developer guide for Multi-Token Prediction (MTP). The documentation is well-structured and provides a good overview of the feature, its usage, and implementation details. I've found a minor but important typo in a command-line example that could cause issues for users. Correcting this will improve the quality of the documentation.
| ## How to Use MTP | ||
| To enable MTP for DeepSeek-V3 models, add the following parameter when starting the service: | ||
|
|
||
| `--speculative_config={"method": "deepseek_mtp", "num_speculative_tokens": 1}'` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a typo in the example command. A trailing single quote ' is present, which will cause a shell error if a user copy-pastes it. Please remove it to ensure the command is correct.
| `--speculative_config={"method": "deepseek_mtp", "num_speculative_tokens": 1}'` | |
| `--speculative_config={"method": "deepseek_mtp", "num_speculative_tokens": 1}` |
Signed-off-by: zouyida2052 <zouyida2002@gmail.com>
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
|
we'll cherry-pick docs from main in one PR later. Thanks. |
What this PR does / why we need it?
add mtp develop guide doc
Does this PR introduce any user-facing change?
no
How was this patch tested?