review: Rerun of PR #1120 (Rascal) on 8xH100 SXM#1177
Open
dexhunter wants to merge 1 commit intoopenai:mainfrom
Open
review: Rerun of PR #1120 (Rascal) on 8xH100 SXM#1177dexhunter wants to merge 1 commit intoopenai:mainfrom
dexhunter wants to merge 1 commit intoopenai:mainfrom
Conversation
Ran the submitted train_gpt.py (commit 39ed402) with SKIP_GPTQ=1 on GCP 8xH100. Result: final_sliding_window_exact val_bpb 1.11350 vs published 1.10979 (seed 300). Gap: +0.00371 BPP — 7x larger than typical seed variance (~0.0005). Note: train_gpt.py contains no quantization code; the published int6+zstd metrics appear to come from an external runner.
newjordan
pushed a commit
to newjordan/parameter-golf-1
that referenced
this pull request
Mar 31, 2026
… script The 2159-line rascal_master (no quantization) was mistakenly committed to records/ instead of the 2468-line script that produced the submission logs. The correct file includes int6+zstd quantization, GPTQ skeleton, and zstandard compression — matching bytes_code=118521 reported in submission.json and logs. Addresses reproducibility concern raised in PR openai#1177. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
sorry man, my agent had replaced the file in git when i was doing optimizations last night. I re-uploaded the proper file. I got my hands in three tests at any given time and it gets messy in my lab. Ive been workign on model quality not wind down, so I had chopped the wind down for my testing. It shoudl not have been pushed. If it will make you feel any better, have an agent scrape my notes and ablations from yesterday and you will have a bunch more data =) i'm working in the open |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Independent rerun of PR #1120 (Rascal, val_bpb 1.1099) on 8xH100 SXM (GCP).
Ran the submitted
train_gpt.pyfrom commit39ed402withSKIP_GPTQ=1, as specified in PR #1120's README reproduction instructions.Rerun Result
final_sliding_window_exact val_bpbfinal_sliding_window_exact val_lossThe rerun
val_bpbis +0.00371 worse than the published seed 300 result. This gap is approximately 7× typical seed variance (~0.0005 std) and 17× the published 3-seed std (0.00021).Environment
a3-highgpu-8g)NCCL_NET=Socket SKIP_GPTQ=1 torchrun --standalone --nproc_per_node=8 train_gpt.pyObservations
The rerun achieves more training steps (6,881 vs 6,593) due to a faster step time (87.2 ms vs ~91 ms), yet the final result is significantly worse.
The submitted
train_gpt.pydoes not contain quantization code. It outputsfinal_model.pt(raw state dict) and computesfinal_sliding_window_exacton the unquantized model. Theint6+zstdquantization andfinal_int6_roundtripmetrics visible in the published seed logs appear to be produced by an external runner rather than bytrain_gpt.pyitself.The reported 3-seed metric (
val_bpb 1.1099) corresponds tofinal_sliding_window_exact, which is measured on the pre-quantization model.Files
RERUN_NOTES.md— detailed notesRERUN_seed1337.log— full rerun output logThis rerun is provided for community transparency, following the precedent of PR #1126 (rerun of PR #1089).