Open
Conversation
ivanl-cerebras
force-pushed
the
il/composite_dataset
branch
from
2916882 to
a67ba6c
Compare
ivanl-cerebras
force-pushed
the
il/composite_dataset
branch
from
a67ba6c to
46d78eb
Compare
Contributor
Author
|
сс @nikolail-cerebras to review |
mklasby
approved these changes
Mar 19, 2026
Contributor
mklasby
left a comment
mklasby
left a comment
There was a problem hiding this comment.
LGTM, minor changes for cleaning up debugger and we can set attention mask in a context manager to make more robust
| tokenizer.save_pretrained(merged_model_dir) | ||
| except Exception as e: | ||
| import pdb; breakpoint() | ||
| import pdb |
| self._current_attention_mask: Optional[torch.Tensor] = None | ||
| super().__init__(model, hook_config) | ||
|
|
||
| def set_attention_mask(self, attention_mask: Optional[torch.Tensor]): |
Contributor
There was a problem hiding this comment.
Rewrite as context manager or manage in a preforwards hook attached to full model
| logger.info("No previous data found @ %s", f_name) | ||
| for sample in tqdm(cat_data, desc=f"Processing {category} samples"): | ||
| model(sample.to(model.device)) | ||
| attention_mask = sample.get("attention_mask", None) |
Contributor
There was a problem hiding this comment.
Suggested change
| attention_mask = sample.get("attention_mask", None) | |
| attn_mask = sample.get("attention_mask", None) | |
| with observer.set_attention_mask(attn_mask) | |
| ... | |
| model(**sample) | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds:
theblackcat102/evol-codealpaca-v1:128,Salesforce/xlam-function-calling-60k:128,open-r1/Mixture-of-Thoughts[code]:128,open-r1/Mixture-of-Thoughts[math]:128,open-r1/Mixture-of-Thoughts[science]:128,SWE-bench/SWE-smith-trajectories(tool):128)Eval results for a trial run
Model: Qwen/Qwen3-30B-A3B
Compression level: 0.25
Dataset: "theblackcat102/evol-codealpaca-v1:8,Salesforce/xlam-function-calling-60k:8,open-r1/Mixture-of-Thoughts[code]:8,open-r1/Mixture-of-Thoughts[math]:8,open-r1/Mixture-of-Thoughts[science]:8,SWE-bench/SWE-smith-trajectories(tool):8"
Batch size: 16
MSL: 2048
HumanEval / HumanEval+: 0.9207 / 0.8780
MBPP / MBPP+: 0.8492 / 0.7222