[Bug]: InternVL3-8B quantize failed: TypeError: can only concatenate str (not "list") to str

[BigFaceBoy]

⚙️ Your current environment

The output of python collect_env.py

Operating System: `Linux-5.4.0-144-generic-x86_64-with-glibc2.31`
Python Version: `3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0]`
llm-compressor Version: `0.8.1`
compressed-tensors Version: `0.12.2`
transformers Version: `4.56.2`
torch Version: `2.8.0`
CUDA Devices: `['NVIDIA L40']`
AMD Devices: `None`

🐛 Describe the bug

I try to quantize InternVL3-8B with lllmcompressor. The Code is:

import base64
from io import BytesIO

import torch
from datasets import load_dataset
from qwen_vl_utils import process_vision_info
from transformers import AutoProcessor, AutoModel

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.utils import dispatch_for_generation

# Load model.
model_id = "/root/InternVL/models/InternVL3-8B"
model = AutoModel.from_pretrained(model_id, torch_dtype="auto", trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Oneshot arguments
DATASET_ID = "/root/InternVL/datasets/flickr30k"
DATASET_SPLIT = {"calibration": "test[:512]"}
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42)


# Apply chat template and tokenize inputs.
def preprocess_and_tokenize(example):
    # preprocess
    buffered = BytesIO()
    example["image"].save(buffered, format="PNG")
    encoded_image = base64.b64encode(buffered.getvalue())
    encoded_image_text = encoded_image.decode("utf-8")
    base64_qwen = f"data:image;base64,{encoded_image_text}"
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": ""},
                {"type": "text", "text": "What does the image show?"},
            ],
        }
    ]


    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    image_inputs, video_inputs = process_vision_info(messages)
    print(image_inputs)

    # tokenize
    return processor(
        text=[text],
        images=image_inputs,
        videos=video_inputs,
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
    )


ds = ds.map(preprocess_and_tokenize, remove_columns=ds["calibration"].column_names)


# Define a oneshot data collator for multimodal inputs.
def data_collator(batch):
    assert len(batch) == 1
    return {key: torch.tensor(value) for key, value in batch[0].items()}


# Recipe
recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W8A8",
        ignore=["lm_head"], # lm_head不量化
    ),
]

# Perform oneshot
oneshot(
    model=model,
    tokenizer=model_id,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    trust_remote_code_model=True,
    data_collator=data_collator,
    sequential_targets=["InternLM2ForCausalLM"],
)



# Save to disk compressed.
SAVE_DIR = "/root/InternVL/models/InternVL3-8B-W8A8"
model.save_pretrained(SAVE_DIR, save_compressed=True)
processor.save_pretrained(SAVE_DIR)

But I got an error:

Traceback (most recent call last):
  File "/root/InternVL/src/w8a8.py", line 65, in <module>
    ds = ds.map(preprocess_and_tokenize, remove_columns=ds["calibration"].column_names)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/datasets/dataset_dict.py", line 946, in map
    dataset_dict[split] = dataset.map(
                          ^^^^^^^^^^^^
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3318, in map
    for rank, done, content in Dataset._map_single(**unprocessed_kwargs):
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3650, in _map_single
    for i, example in iter_outputs(shard_iterable):
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3624, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3547, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/InternVL/src/w8a8.py", line 48, in preprocess_and_tokenize
    text = processor.apply_chat_template(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1640, in apply_chat_template
    rendered_chat, generation_indices = render_jinja_template(
                                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/transformers/utils/chat_template_utils.py", line 521, in render_jinja_template
    rendered_chat = compiled_template.render(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
  File "/root/miniforge3/envs/quant/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 23, in top-level template code
TypeError: can only concatenate str (not "list") to str

🛠️ Steps to reproduce

No response

[brian-dellabetta]

Hi @BigFaceBoy , this error appears to occur in the data loading and preprocessing stage, before anything from llm-compressor is invoked. Something must be wrong with your usage of datasets and the model. You might be able to figure out how to resolve from the details in the model card here

[BigFaceBoy]

@brian-dellabetta Is there any sample for InternVL3 quantization with llmcompressor?

[kylesayrs]

@BigFaceBoy LLM Compressor doesn't currently have any examples of InternVL3. However, community contributions of examples for this model are appreciated!

[BigFaceBoy]

update: Soveld the problem by check the chat_template.jinja .
But I got the same error as 1929 . It shows that the loss field is None , but I have no idea what to do next to resolve is?

[alew3]

I think this is unrelated to llm compressor, as I'm also getting this error when running vllm with OpenGVLab/InternVL3-1B-Instruct and calling the model.

[brian-dellabetta]

update: Soveld the problem by check the chat_template.jinja . But I got the same error as 1929 . It shows that the loss field is None , but I have no idea what to do next to resolve is?

@BigFaceBoy can you post your resolution to chat_template.jinja so others can see? Try running oneshot with pipeline="basic" to circumvent the sequential onloading issue you're hitting further down

[BigFaceBoy]

@brian-dellabetta I download the chat_template.jinja from InternVL3_5-8B and place the file in the local dir of InternVL3-8B. The content of chat_template.jinja is:

{% for message in messages %}{{'<|im_start|>' + message['role'] + '
'}}{% if message['content'] is string %}{{ message['content'] }}{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' %}{{ '<image>
' }}{% elif content['type'] == 'video' %}{{ '<video>
' }}{% elif content['type'] == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}{% endif %}{{'<|im_end|>
'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant
' }}{% endif %}

I will merge the example of quantizing InternVL to llmcompressor