'Nonetype' errors when running taco, livecodebench, apps tasks.

[ZijunSong]

Sorry, when I use

skythought evaluate \
	--model /data3/models/hub/models--google--gemma-2-2b-it/snapshots/299a8560bedf22ed1c72a8a11e7dce4a7f9f51f8 \
	--task taco

to run taco task, I meet an error

INFO 03-13 15:35:58 __init__.py:183] Automatically detected platform cuda.
INFO:skythought.evals.cli:Temperature: 0, top_p: 1.0, max_tokens: 32768
/data3/workhome/songzijun/SkyThought/skythought/evals/cli.py:290: UserWarning: Model /data3/models/hub/models--google--gemma-2-2b-it/snapshots/299a8560bedf22ed1c72a8a11e7dce4a7f9f51f8 not found in /data3/workhome/songzijun/SkyThought/skythought/evals/models/model_configs.yaml. Initializing without any system prompt.
  model_config = ModelConfig.from_model_id(
Traceback (most recent call last):
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/bin/skythought", line 8, in <module>
    sys.exit(main())
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/cli.py", line 563, in main
    app()
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/main.py", line 340, in __call__
    raise e
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/main.py", line 323, in __call__
    return get_command(self)(*args, **kwargs)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/core.py", line 743, in main
    return _main(
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/core.py", line 198, in _main
    rv = self.invoke(ctx)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/main.py", line 698, in wrapper
    return callback(**use_params)
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/cli.py", line 321, in evaluate
    generate_and_score(
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/inference_and_check.py", line 372, in generate_and_score
    eval_data = handler.load_and_filter_dataset(start, end)
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/tasks/taco/taco_handler.py", line 97, in load_and_filter_dataset
    dataset = self.load_dataset(subset=subset, split=split).to_pandas()
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/tasks/base.py", line 85, in load_dataset
    dataset = load_dataset(
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/datasets/load.py", line 1886, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable

and I don't know how to fix it.
Besides, I meet the same error when running livecodebench and apps tasks. Thank you for your help!

[SumanthRH]

Hi @ZijunSong are you using the latest changes from main ? I'm unable to reproduce this locally.

If you're facing the issues even with the latest changes, it would be helpful to know more about your environment:

1. Platform (Linux/ MacoS / etc)
2. Hardware
3. Python environment details: versions of torch, huggingface datasets, vllm, etc (We recommend starting from a fresh environment with python >= 3.10 and following the official installation instructions)

[SumanthRH]

@ZijunSong do you still have this issue?

[ZijunSong]

Yes, I'm sorry I still have this issue. After downloading the latest code, when I use

conda activate skybench && cd SkyThought
export HF_ENDPOINT=https://hf-mirror.com
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
skythought evaluate \
	--model /share_data/data2/workhome/chenhaotian/models/models--Team-ACE--ToolACE-2-8B/snapshots/91c918927e687541cc3f8dd4dbed42014efbc147 \
	--backend vllm \
	--backend-args tensor_parallel_size=8 \
	--task taco \
	--overwrite

I still meet the same error in the specific datasets.

INFO 03-25 08:03:18 __init__.py:183] Automatically detected platform cuda.
INFO:skythought.evals.cli:Temperature: 0, top_p: 1.0, max_tokens: 32768
/data3/workhome/songzijun/SkyThought/skythought/evals/cli.py:301: UserWarning: Model /share_data/data2/workhome/chenhaotian/models/models--Team-ACE--ToolACE-2-8B/snapshots/91c918927e687541cc3f8dd4dbed42014efbc147 not found in /data3/workhome/songzijun/SkyThought/skythought/evals/models/model_configs.yaml. Initializing without any system prompt.
  model_config = ModelConfig.from_model_id(
Traceback (most recent call last):
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/bin/skythought", line 8, in <module>
    sys.exit(main())
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/cli.py", line 586, in main
    app()
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/main.py", line 340, in __call__
    raise e
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/main.py", line 323, in __call__
    return get_command(self)(*args, **kwargs)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/core.py", line 743, in main
    return _main(
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/core.py", line 198, in _main
    rv = self.invoke(ctx)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/typer/main.py", line 698, in wrapper
    return callback(**use_params)
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/cli.py", line 332, in evaluate
    generate_and_score(
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/inference_and_check.py", line 406, in generate_and_score
    eval_data = handler.load_and_filter_dataset(start, end)
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/tasks/taco/taco_handler.py", line 91, in load_and_filter_dataset
    dataset = self.load_dataset(subset=subset, split=split).to_pandas()
  File "/data3/workhome/songzijun/SkyThought/skythought/evals/tasks/base.py", line 85, in load_dataset
    dataset = load_dataset(
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/data3/workhome/songzijun/miniconda3/envs/skybench/lib/python3.10/site-packages/datasets/load.py", line 1886, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable

Specific configurations are as follows:

Linux pm-ac42 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Python 3.10.16

Name: torch
Version: 2.5.1

Name: datasets
Version: 3.3.2

Name: vllm
Version: 0.7.0

Thank you for your help!

[SumanthRH]

@ZijunSong My apologies for the delay. Thanks for providing more details! I can now reproduce this issue for TACO with your settings

Python 3.10.16

Name: torch
Version: 2.5.1

Name: datasets
Version: 3.3.2

Name: vllm
Version: 0.7.0

Min Reproduction

import datasets
dataset = datasets.load_dataset("BAAI/TACO", "MEDIUM", split="train",  trust_remote_code=True, download_mode="force_redownload")  # ignore cache and redownload the dataset

Can you use datasets==2.21.0 and see if that resolves the issue?

Edit: I forgot to mention, this is all with HF_ENDPOINT=https://hf-mirror.com

[SumanthRH]

@ZijunSong is this issue resolved now?

I am able to download the dataset without issues if I unset HF_ENDPOINT right now.

It's only with HF_ENDPOINT=https://hf-mirror.com where I still see some issues with datasets>=3.0.0. For this particular endpoint, you can use datasets==2.21.0