Issues exist with the settings for num generation and the number of model-generated responses (Completion). The number of Completion responses consistently equals per_device_batch_size multiplied by gradient_acc_steps.

[Haoyong23]

During runtime, I must set per_device_batch_size*gradient_acc_steps equal to num generation. Otherwise, the number of responses per sample (Completions) will be per_device_batch_size*gradient_acc_steps, while the total steps will change.
For example: Running on 4 GPUs with per_device_batch_size=1, gradient_acc_steps=8, and num_generation=8, each sample outputs 8 responses. The total steps become Step=203num_generation/per_device_batch_size/gradient_acc_steps=15.

Keeping everything else constant, changing gradient_acc_steps to 4 results in 4 completions per sample being printed. This equals per_device_batch_size * gradient_acc_steps and no longer equals num_generation (8). However, the total steps change from 15 to 30. The total number of samples processed remains unchanged, and I'm unsure how this affects training.

Changing gradient_acc_steps to 16 should theoretically yield total steps of 203num_generation/per_device_batch_size/gradient_acc_steps=7.5. However, actual runs show total steps as 6, while Completion remains per_device_batch_size*gradient_acc_steps, outputting 16.

I looked up resources, but they only state that (number of GPUs * batch size * gradient) % num_generations must be ensured.

Does anyone know why this happens? To ensure each prompt generates 8 responses for training, must I set per_device_batch_size=1, gradient_acc_steps=8, and num_generation=8? Will this configuration guarantee accuracy? What are the limitations of other settings? Or is it irrelevant? Is it sufficient to only ensure (number_of_cards * batch_size * gradient) % num_generation?

[qgallouedec]

Thanks for reporting. Could you please try to phrase your question way more clearly? At the moment, it is incomprehensible.

It seems that you want to set the number of completions per prompt. To do this, you can just use the num_generations parameter.

[Haoyong23]

I apologize for the confusion caused by my unclear explanation. Let me clarify the issue further: 1. I expect each prompt to generate 8 responses for GRPO reinforcement training; 2. I have set `num_generation=8`. Since I want to test whether the code runs correctly with a small sample first, I used `print` to output the `Completions` list to observe the multiple responses returned by the model and the Reward values; 3. During printing, I discovered an issue: the number of elements in the `Completions` and `Reward` lists does not consistently match `num_generation`. Specifically, during testing, I observed that the number of elements in both the Completions and Reward lists consistently matched per_device_batch_size multiplied by gradient_acc_steps. For instance, with per_device_batch_size=1, gradient_acc_steps=4, and num_generation=8, the lists contained exactly 4 elements each. 4. Building on this, I conducted further tests. When per_device_batch_size=1, gradient_acc_steps=16, and num_generation=8, the number of elements in the Completions and Reward lists was 16. Does this imply that the number of elements in these lists always equals per_device_batch_size * gradient_acc_steps? 5. After observing this phenomenon, I further investigated whether the total training sample count changed. Logically, the total training sample count = number of dataset entries * number of epochs * num_generation. The training step count = total training sample count / per_device_batch_size / gradient_acc_steps / number of GPUs. Using the control variable method, I only altered the gradient_acc_steps value. 6. For this test, I set dataset size to 20, training epochs to 3, GPU count to 4, per_device_batch_size=1, and num_generation=8. (1) When gradient_acc_steps=4, training steps=30; (2) When gradient_acc_steps=8, training steps=15; (3) When gradient_acc_steps=16, training Step=6; Based on my inference, (3) should yield 7.5, rounded to 8. Therefore, I summarize my questions as follows: (1) Does the number of generated responses included in the Completions returned by the GRPO algorithm consistently match per_device_batch_size*gradient_acc_steps? What is the core significance of the num_generation value setting? (2) I currently expect each prompt to return 8 generated responses. However, based on my current testing, only when both `per_device_batch_size*gradient_acc_steps` and `num generation` are set to 8 does the output superficially match my expectation. I still do not understand the generation logic of `Completions`. (3) When num_generation is set to 8 but per_device_batch_size*gradient_acc_steps has other values, the number of elements in the Completions and Reward return lists no longer equals 8. Could this potentially impact my training? (4) In the scenario described in test (3), why does the training step count equal 6 when gradient_acc_steps=16? The above provides a more detailed explanation of the issue. I look forward to your response and wish you success in your research! 在 2025-10-27 22:43:43，"Quentin Gallouédec" ***@***.***> 写道： qgallouedec left a comment (huggingface/trl#4346) Thanks for reporting. Could you please try to phrase your question way more clearly? At the moment, it is incomprehensible. It seems that you want to set the number of completions per prompt. To do this, you can just use the num_generations parameter. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[qgallouedec]

It's because we generate per batch of prompts, not per prompt, to be more efficient. Check the prompts, you'll see that the number of unique prompts matches the num_generations

[Haoyong23]

Thank you very much for your reply. Does this mean that setting num_generations=8 will meet my training requirements? 在 2025-10-27 23:45:31，"Quentin Gallouédec" ***@***.***> 写道： qgallouedec left a comment (huggingface/trl#4346) It's because we generate per batch of prompts, not per prompt, to be more efficient. Check the prompts, you'll see that the number of unique prompts matches the num_generations — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[qgallouedec]

Yes

[Haoyong23]

Thank you for your patient response. I have clearly understood. Wishing you all the best. 在 2025-10-28 03:15:38，"Quentin Gallouédec" ***@***.***> 写道： qgallouedec left a comment (huggingface/trl#4346) Yes — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>