Extend max_model_length to prevent context truncation

[eldarkurtic]

With the existing example command for evaluation, both max_model_length and max_new_tokens are set to the same value of 32768. This produces the following warning:

[ WARNING]: context_size + max_new_tokens=33238 which is greater than self.max_length=32768. Truncating context to 0 tokens. (vllm_model.py:276)

We need to increase the max_model_length to a larger value to prevent truncating the context. I ran some tests and the proposed 38768 seems to be large enough for all three tasks I've tested so far: AIME, MATH-500, and GPQA-Diamond.

[lewtun]

Thanks for the fix @eldarkurtic! Do you see much variation in the evaluation scores once this is included? I believe some of the Qwen models were trained with 32k context, so once we exceed this the model may start producing gibberish.

[eldarkurtic]

I've never done a full eval to compare the two because I stopped using 32k as soon as I saw the warning that prompt has been truncated to 0 tokens

[ WARNING]: context_size + max_new_tokens=33238 which is greater than self.max_length=32768. Truncating context to 0 tokens. (vllm_model.py:276)

But definitely a good point for Qwen models. I assume this could be up to user to decide based on their chosen model for evals.