creating artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model

[virtustate]

This is very cool, thanks for creating and sharing! From all the .md files, it looks like you used some pretty sophisticated AI assisted coding. I'd love to hear what you used and any details you'd like to share.

Some scripts, e.g. scripts/data/run_sample.sh, fail, not finding the tokenizer: artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model which I have yet to figure out how to create.

scripts/data/run_full.sh looks promising to create the tokenizer, but failed finding a training split:
Bad split: train. Available splits: ['test']

Any hints on getting farther with this appreciated. Thanks!

1410 │ │ elif split in splits_generators: │
│ 1411 │ │ │ splits_generator = splits_generators[split] │
│ 1412 │ │ else: │
│ ❱ 1413 │ │ │ raise ValueError(f"Bad split: {split}. Available splits: {list(splits_genera │
│ 1414 │ │ │
│ 1415 │ │ # Create a dataset for each of the given splits │
│ 1416 │ │ datasets = map_nested( │
│ │
│ ╭───────────────────────────────────────────────────────────── locals ──────────────────────────────────────────────────────────────╮ │
│ │ base_path = None │ │
│ │ dl_manager = <datasets.download.streaming_download_manager.StreamingDownloadManager object at 0x1156d53a0> │ │
│ │ self = <datasets.packaged_modules.text.text.Text object at 0x1068c5970> │ │
│ │ split = 'train' │ │
│ │ splits_generators = { │ │
│ │ │ 'test': SplitGenerator( │ │
│ │ │ │ name='test', │ │
│ │ │ │ gen_kwargs={ │ │
│ │ │ │ │ 'files': [ │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x11577e630>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x114991e80>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x11563e2a0>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x116a6f260>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x116a6f920>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x116a6eb70>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x116a6f0b0>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x116a6f7a0>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x116a6ec30>, │ │
│ │ │ │ │ │ <datasets.utils.file_utils.FilesIterable object at 0x116a6ee70>, │ │
│ │ │ │ │ │ ... +92 │ │
│ │ │ │ │ ] │ │
│ │ │ │ }, │ │
│ │ │ │ split_info=SplitInfo( │ │
│ │ │ │ │ name='test', │ │
│ │ │ │ │ num_bytes=0, │ │
│ │ │ │ │ num_examples=0, │ │
│ │ │ │ │ shard_lengths=None, │ │
│ │ │ │ │ dataset_name=None │ │
│ │ │ │ ) │ │
│ │ │ ) │ │
│ │ }
ValueError: Bad split: train. Available splits: ['test']

[neelandspaul]

Could not find the file: artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model

[neelandspaul]

What is the format of the file |:artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model ?

[kmccleary3301]

Thanks for the kind words and for bringing this up. Quick answers below.

AI assistance I used

• I used GPT‑5.1‑Pro to help interpret tricky parts of the HOPE/Nested Learning paper and turn them into concrete specs; it also caught a few inconsistencies I missed on the first pass.
• I used Codex for spec drafting, alignment, and expediting tests/boilerplate. I still do a fair amount manually, and I stick to workflows I’ve already vetted on research projects like these.
• No proprietary weights or closed assets are shipped—everything in the repo is plain code/config/docs.

Fixing the tokenizer + run scripts

Easiest path: sample pipeline (recommended first)

Run:

uv run bash scripts/data/run_sample.sh

This will:

• Download the small “sample” mixture (configs/data/refinedweb_mixture_sample.yaml).
• Train the tokenizer (32k unigram) and write it to artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model.
• Shard the sample corpus and write mixture stats.

After this, scripts looking for artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model should succeed.

Full pipeline notes

Run:

uv run bash scripts/data/run_full.sh

If you hit Bad split: train. Available splits: ['test'], that means the HF dataset you’re streaming only exposes test. You can:

• Set SPLIT=test in your environment, or
• Edit the relevant entry in configs/data/refinedweb_mixture_full.yaml to use split: test for that source.

You can also cap downloads with RW_LIMIT / NUM_LINES env vars (see the script header).

Just training the tokenizer

If you already have text locally:

uv run python scripts/data/train_tokenizer.py \
  --input data/filtered/refinedweb_sample.txt \
  --vocab-size 32000 \
  --output-dir artifacts/tokenizer/refinedweb_mix \
  --model-prefix spm_32000_unigram

Adjust --input to your file(s). This will create artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model so other scripts stop complaining.

Common gotchas

• Ensure artifacts/tokenizer/refinedweb_mix/ exists (scripts will create it if missing).
• If a corpus doesn’t offer train, switch to test in the mixture YAML or export SPLIT=test.
• Validate end-to-end with the sample pipeline first before attempting the full mixture.

I can probably just push a small patch to auto-fallback to test in run_full.sh and add a short README snippet if that would help.

[hustcselwb]

Thanks for the kind words and for bringing this up. Quick answers below.

AI assistance I used

• I used GPT‑5.1‑Pro to help interpret tricky parts of the HOPE/Nested Learning paper and turn them into concrete specs; it also caught a few inconsistencies I missed on the first pass.
• I used Codex for spec drafting, alignment, and expediting tests/boilerplate. I still do a fair amount manually, and I stick to workflows I’ve already vetted on research projects like these.
• No proprietary weights or closed assets are shipped—everything in the repo is plain code/config/docs.

Fixing the tokenizer + run scripts

Easiest path: sample pipeline (recommended first)

Run:

uv run bash scripts/data/run_sample.sh
This will:

• Download the small “sample” mixture (configs/data/refinedweb_mixture_sample.yaml).
• Train the tokenizer (32k unigram) and write it to artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model.
• Shard the sample corpus and write mixture stats.

After this, scripts looking for artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model should succeed.

Full pipeline notes

Run:

uv run bash scripts/data/run_full.sh
If you hit Bad split: train. Available splits: ['test'], that means the HF dataset you’re streaming only exposes test. You can:

• Set SPLIT=test in your environment, or
• Edit the relevant entry in configs/data/refinedweb_mixture_full.yaml to use split: test for that source.

You can also cap downloads with RW_LIMIT / NUM_LINES env vars (see the script header).

Just training the tokenizer

If you already have text locally:

uv run python scripts/data/train_tokenizer.py
--input data/filtered/refinedweb_sample.txt
--vocab-size 32000
--output-dir artifacts/tokenizer/refinedweb_mix
--model-prefix spm_32000_unigram
Adjust --input to your file(s). This will create artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model so other scripts stop complaining.

Common gotchas

• Ensure artifacts/tokenizer/refinedweb_mix/ exists (scripts will create it if missing).
• If a corpus doesn’t offer train, switch to test in the mixture YAML or export SPLIT=test.
• Validate end-to-end with the sample pipeline first before attempting the full mixture.

I can probably just push a small patch to auto-fallback to test in run_full.sh and add a short README snippet if that would help.

Thank you so much for your contribution and hard work! I have a question: is the implementation of Titans in this library the same as the implementation in this library: https://github.com/lucidrains/titans-pytorch? It seems to me that the Titans implementation in your repository is more concise?

[mdengler]

run_sample.sh doesn't work for me, unfort; I'm getting "not found" for the .model file still:

20251207-23:36.12 martin@spark nested_learning ▶ $ LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/nvshmem/13/ uv run bash scripts/data/run_sample.sh        
[Data] Creating filtered RefinedWeb sample                                                                                                                                                    
[...]
Repo card metadata block was not found. Setting CardData to empty.
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 53/53 [00:00<00:00, 217.31it/s]
Filtering dataset: 1216it [00:04, 300.81it/s]
[Filter] Processed=1217 kept=1000 -> data/filtered/code_en_sample.txt
[Data] Sharding filtered samples
[...]
OSError: Not found: "artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model": No such file or directory Error #2

[mdengler]

This project is awesome and I'd love to see it work out-of-the-box; I can definitely suggest some PRs but I first am just assuming I'm doing something wrong.

Is the current git repo run_full.sh working as-committed? I have to adjust the splits (because there are no train splits), but then I still get an error about the vocabulary size :(

RuntimeError: Internal: src/trainer_interface.cc(664) [(trainer_spec_.vocab_size()) == (model_proto->pieces_size())] Vocabulary size too high (32000). Please set it to a value <= 2056.

The train-split patch/changes and full output attached:

run_log-20251208101940.txt

[hustcselwb]

run_sample.sh很遗憾，对我来说还是不行；我仍然收到“找不到.model文件”的错误提示：

20251207-23:36.12 martin@spark nested_learning ▶ $ LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/nvshmem/13/ uv run bash scripts/data/run_sample.sh        
[Data] Creating filtered RefinedWeb sample                                                                                                                                                    
[...]
Repo card metadata block was not found. Setting CardData to empty.
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 53/53 [00:00<00:00, 217.31it/s]
Filtering dataset: 1216it [00:04, 300.81it/s]
[Filter] Processed=1217 kept=1000 -> data/filtered/code_en_sample.txt
[Data] Sharding filtered samples
[...]
OSError: Not found: "artifacts/tokenizer/refinedweb_mix/spm_32000_unigram.model": No such file or directory Error #2

I encountered this problem before. You should train the tokenizer first and then run_sample.sh again.

[JetRichardLee]

I also encouter the Bad split: train. Available splits: ['test'] but I think it's just load a wrong dataset in the write_sample (loaded a dataset named "text"). I found the way to solve this by making several modifications. The main issue seems to come from the wrong setting of yaml and the _write_sample.

To fix this, first change /configs/data/refinedweb_mixture.yaml to:

name: refinedweb_mix_v1
tokenizer_output_dir: artifacts/tokenizer/refinedweb_mix
datasets:
  - name: refinedweb
    dataset: text
    split: train
    text_column: text
    path: HuggingFaceFW/fineweb
    data_files: data/filtered/refinedweb_en_full.txt
    seq_len: 2048
    sequences_per_shard: 2048
    output_dir: data/shards/refinedweb
    max_records: null
  - name: books
    dataset: text
    split: train
    text_column: text
    path: wikimedia/wikipedia
    subset: 20231101.en
    data_files: data/filtered/wikipedia_en_full.txt
    seq_len: 2048
    sequences_per_shard: 2048
    output_dir: data/shards/wikipedia
    max_records: null
  - name: c4
    dataset: text
    split: train
    text_column: text
    path: allenai/c4
    subset: en
    data_files: data/filtered/c4_en_full.txt
    seq_len: 2048
    sequences_per_shard: 2048
    output_dir: data/shards/c4
    max_records: null
  - name: redpajama
    dataset: text
    split: train
    text_column: text
    path: cerebras/SlimPajama-627B
    data_files: data/filtered/redpajama_en_full.txt
    seq_len: 2048
    sequences_per_shard: 2048
    output_dir: data/shards/redpajama
    max_records: null
  - name: code
    dataset: text
    split: train
    text_column: text
    path: codeparrot/codeparrot-clean-train
    data_files: data/filtered/code_en_full.txt
    seq_len: 2048
    sequences_per_shard: 2048
    output_dir: data/shards/code
    max_records: null

This will lead them to the right path in load_dataset. All datasets provided now have train spilt in HF.

Then you will need also to change the scripts/data/train_tokenizer.py to load these paths correctly:

@dataclass
class DatasetSpec:
    name: str
    dataset: str
    path: str
    split: str = "train"
    subset: str | None = None
    text_column: str = "text"
    sample_limit: int = 100_000


def _load_specs_from_manifest(manifest: Path) -> List[DatasetSpec]:
    data = yaml.safe_load(manifest.read_text())
    entries = data.get("datasets", data)
    specs = []
    for entry in entries:
        specs.append(
            DatasetSpec(
                name=entry.get("name") or entry["dataset"].split("/")[-1],
                path = entry["path"],
                dataset=entry["dataset"],
                split=entry.get("split", "train"),
                subset=entry.get("subset"),
                text_column=entry.get("text_column", "text"),
                sample_limit=entry.get("sample_limit", 100_000),
            )
        )
    return specs


def _write_samples(spec: DatasetSpec, handle) -> int:
    #ds = load_dataset(spec.dataset, spec.subset, split=spec.split, streaming=True)
    ds = load_dataset(spec.path,spec.subset,split=spec.split,streaming=True)

    count = 0
    for row in ds:
        text = row.get(spec.text_column)
        if not isinstance(text, str):
            continue
        handle.write(text.replace("\n", " ") + "\n")
        count += 1
        if spec.sample_limit > 0 and count >= spec.sample_limit:
            break
    return count

Eventually, you could train the tokenizer first successfully and then run the example bash.