Packing with VLMs

[jiosephlee]

Feature request

Per axolotl-ai-cloud/axolotl#3131, would it be possible to implement packing (in SFT) for VLMs? This is currently disabled in SFTrainer by reading the class of the provided processor.

Motivation

This would be super helpful for reducing the memory footprint of training VLMs.

Your contribution

If everything is processed into tokens, shouldn't packing with other modalities be straightforward (ignoring memory issues with the vision encoder)? I'm not knowledgeable in how padding_free packing interacts with flash attention 2. Is this the reason it is not straightforward?

[qgallouedec]

It's pretty tricky to implement for VLM, because data is processed on the fly.

[marseller]

It's pretty tricky to implement for VLM, because data is processed on the fly.

In this case what would be the most straight forward way of using packing with a VLM like gemma3 for sft if we are interested only in fine-tuning with text data?