. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n"
- ]
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SXd9bTZd1aaL"
+ },
+ "source": [
+ "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"
+ ]
},
{
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "0aa70e83519f45ea961d49258c3fe2c2",
- "version_major": 2,
- "version_minor": 0
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "6bZsfBuZDeCL"
},
- "text/plain": [
- "Downloading readme: 0%| | 0.00/442 [00:00 0 ! Suggested 8, 16, 32, 64, 128\n",
+ " target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
+ " \"gate_proj\", \"up_proj\", \"down_proj\",],\n",
+ " lora_alpha = 16,\n",
+ " lora_dropout = 0, # Supports any, but = 0 is optimized\n",
+ " bias = \"none\", # Supports any, but = \"none\" is optimized\n",
+ " # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",
+ " use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n",
+ " random_state = 3407,\n",
+ " use_rslora = False, # We support rank stabilized LoRA\n",
+ " loftq_config = None, # And LoftQ\n",
+ ")"
]
- },
- "metadata": {},
- "output_type": "display_data"
},
{
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "90dc506f8ecc4cd9b696cf64e2d90153",
- "version_major": 2,
- "version_minor": 0
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vITh0KVJ10qX"
},
- "text/plain": [
- "Downloading data: 0%| | 0.00/8.24M [00:00\n",
+ "### Data Prep\n",
+ "We now use the `ChatML` format for conversation style finetunes. We use [Open Assistant conversations](https://huggingface.co/datasets/philschmid/guanaco-sharegpt-style) in ShareGPT style. ChatML renders multi turn conversations like below:\n",
+ "\n",
+ "```\n",
+ "<|im_start|>system\n",
+ "You are a helpful assistant.<|im_end|>\n",
+ "<|im_start|>user\n",
+ "What's the capital of France?<|im_end|>\n",
+ "<|im_start|>assistant\n",
+ "Paris.\n",
+ "```\n",
+ "\n",
+ "**[NOTE]** To train only on completions (ignoring the user's input) read TRL's docs [here](https://huggingface.co/docs/trl/sft_trainer#train-on-completions-only).\n",
+ "\n",
+ "This script uses the chatml chat template to format ShareGPT-style conversations for training.\n",
+ "\n",
+ "Unlike Unsloth's get_chat_template() wrapper, we manually map\n",
+ "\n",
+ "`{\"from\": \"human\"}` and `{\"from\": \"gpt\"}`\n",
+ "\n",
+ "to the mistral standard\n",
+ "\n",
+ "`{\"role\": \"user\"}` and `{\"role\": \"assistant\"}`.\n",
+ "\n",
+ "For text completions like novel writing, try this [notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb)."
]
- },
- "metadata": {},
- "output_type": "display_data"
},
{
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "17e5228c2f7d44d3a25489767887c164",
- "version_major": 2,
- "version_minor": 0
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 49,
+ "referenced_widgets": [
+ "16a897b9e4f44c059727761149c8b60b",
+ "b80060c928c74867bde1123f4d6c9ef8",
+ "00a331509c244881962ba2175adcaab8",
+ "b994df4c402b4924b5afaba411170041",
+ "ce755f8341934515a487433fc98ac610",
+ "efb28e4a2e9b4aaf956c262d8c0cffc7",
+ "70de9ffc8e324938924c9c64ed98e33e",
+ "9a3113cd35634ce58d714d3e519f7d6b",
+ "58e7e14f6f304c78954c8424513785b2",
+ "8985687193174cd0990fbcbbb3385102",
+ "92bfd6194ae6467c83edd96140ca412b"
+ ]
+ },
+ "id": "LjY75GoYUCB8",
+ "outputId": "b1d437ba-7add-4649-d04a-b3fc6a77eeb3"
},
- "text/plain": [
- "Generating train split: 0%| | 0/9033 [00:00 to instead\n",
- ")\n",
- "\n",
- "def formatting_prompts_func(examples):\n",
- " convos = examples[\"conversations\"]\n",
- " texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]\n",
- " return { \"text\" : texts, }\n",
- "pass\n",
- "\n",
- "from datasets import load_dataset\n",
- "dataset = load_dataset(\"philschmid/guanaco-sharegpt-style\", split = \"train\")\n",
- "dataset = dataset.map(formatting_prompts_func, batched = True,)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "cHiVoToneynS"
- },
- "source": [
- "Let's see how the `ChatML` format works by printing the 5th element"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
},
- "id": "4GSuKSSbpYKq",
- "outputId": "a21a2006-2c8d-4974-e41a-d35a7ec2f5de"
- },
- "outputs": [
{
- "data": {
- "text/plain": [
- "[{'from': 'human',\n",
- " 'value': 'What is the typical wattage of bulb in a lightbox?'},\n",
- " {'from': 'gpt',\n",
- " 'value': 'The typical wattage of a bulb in a lightbox is 60 watts, although domestic LED bulbs are normally much lower than 60 watts, as they produce the same or greater lumens for less wattage than alternatives. A 60-watt Equivalent LED bulb can be calculated using the 7:1 ratio, which divides 60 watts by 7 to get roughly 9 watts.'},\n",
- " {'from': 'human',\n",
- " 'value': 'Rewrite your description of the typical wattage of a bulb in a lightbox to only include the key points in a list format.'}]"
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "4GSuKSSbpYKq",
+ "outputId": "39811591-d1ec-4f8a-8945-4629e087c692"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "{'role': 'user', 'content': 'What is the typical wattage of bulb in a lightbox?'}\n",
+ "{'role': 'assistant', 'content': 'The typical wattage of a bulb in a lightbox is 60 watts, although domestic LED bulbs are normally much lower than 60 watts, as they produce the same or greater lumens for less wattage than alternatives. A 60-watt Equivalent LED bulb can be calculated using the 7:1 ratio, which divides 60 watts by 7 to get roughly 9 watts.'}\n",
+ "{'role': 'user', 'content': 'Rewrite your description of the typical wattage of a bulb in a lightbox to only include the key points in a list format.'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "for msg in dataset[5][\"messages\"]:\n",
+ " print(\"{'role': '\" + msg['role'] + \"', 'content': '\" + msg['content'] + \"'}\")"
]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "dataset[5][\"conversations\"]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
},
- "id": "U5iEWrUkevpE",
- "outputId": "bfea084f-d9f9-4373-9ead-4b2b4380e714"
- },
- "outputs": [
{
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "<|im_start|>user\n",
- "What is the typical wattage of bulb in a lightbox?<|im_end|>\n",
- "<|im_start|>assistant\n",
- "The typical wattage of a bulb in a lightbox is 60 watts, although domestic LED bulbs are normally much lower than 60 watts, as they produce the same or greater lumens for less wattage than alternatives. A 60-watt Equivalent LED bulb can be calculated using the 7:1 ratio, which divides 60 watts by 7 to get roughly 9 watts.<|im_end|>\n",
- "<|im_start|>user\n",
- "Rewrite your description of the typical wattage of a bulb in a lightbox to only include the key points in a list format.<|im_end|>\n",
- "\n"
- ]
- }
- ],
- "source": [
- "print(dataset[5][\"text\"])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "GuKOAUDpUeDL"
- },
- "source": [
- "If you're looking to make your own chat template, that also is possible! You must use the Jinja templating regime. We provide our own stripped down version of the `Unsloth template` which we find to be more efficient, and leverages ChatML, Zephyr and Alpaca styles.\n",
- "\n",
- "More info on chat templates on [our wiki page!](https://github.com/unslothai/unsloth/wiki#chat-templates)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "p31Z-S6FUieB"
- },
- "outputs": [],
- "source": [
- "unsloth_template = \\\n",
- " \"{{ bos_token }}\"\\\n",
- " \"{{ 'You are a helpful assistant to the user\\n' }}\"\\\n",
- " \"{% for message in messages %}\"\\\n",
- " \"{% if message['role'] == 'user' %}\"\\\n",
- " \"{{ '>>> User: ' + message['content'] + '\\n' }}\"\\\n",
- " \"{% elif message['role'] == 'assistant' %}\"\\\n",
- " \"{{ '>>> Assistant: ' + message['content'] + eos_token + '\\n' }}\"\\\n",
- " \"{% endif %}\"\\\n",
- " \"{% endfor %}\"\\\n",
- " \"{% if add_generation_prompt %}\"\\\n",
- " \"{{ '>>> Assistant: ' }}\"\\\n",
- " \"{% endif %}\"\n",
- "unsloth_eos_token = \"eos_token\"\n",
- "\n",
- "if False:\n",
- " tokenizer = get_chat_template(\n",
- " tokenizer,\n",
- " chat_template = (unsloth_template, unsloth_eos_token,), # You must provide a template and EOS token\n",
- " mapping = {\"role\" : \"from\", \"content\" : \"value\", \"user\" : \"human\", \"assistant\" : \"gpt\"}, # ShareGPT style\n",
- " map_eos_token = True, # Maps <|im_end|> to instead\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "idAEIeSQ3xdS"
- },
- "source": [
- "\n",
- "### Train the model\n",
- "Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support TRL's `DPOTrainer`!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 122,
- "referenced_widgets": [
- "f50eabd823c540dbaf4f3dc49364093d",
- "f9e39ce10a2c448dac480bad52f38551",
- "283e030dbc4d4e62bf85f801f717f733",
- "fe1c443b725f4b35bb68ece80813d295",
- "740b55e25f2441ea82f075a0f5ca4d89",
- "4a83df074c3e48099de90a71ba45e389",
- "644af20034774c58bfc85e8dfa07ef2b",
- "230306061b6f45c4967ee6e1ab02449f",
- "f0ad65eda4ad4fc18b7458293196cfd4",
- "11a52380663f4514806f48205e7f18a5",
- "b2a21adebbd9441faa161d98e2527790"
- ]
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "GuKOAUDpUeDL"
+ },
+ "source": [
+ "Now we can convert this data to what the Mistral v0.3 tokenizer expects.\n",
+ "\n",
+ "If you're looking to make your own chat template, that also is possible! You must use the Jinja templating regime.\n",
+ "\n",
+ "More info on chat templates on [our wiki page!](https://github.com/unslothai/unsloth/wiki#chat-templates)"
+ ]
},
- "id": "95_Nn-89DhsL",
- "outputId": "b6d665ab-f31a-482e-9bab-10f2819c7186"
- },
- "outputs": [
{
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/multiprocess/popen_fork.py:66: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.\n",
- " self.pid = os.fork()\n"
- ]
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "p31Z-S6FUieB",
+ "outputId": "a6ada8e5-8ede-4788-bcf1-a4dca1c97b05",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 87,
+ "referenced_widgets": [
+ "ea2d16b0260a4530b35c391f6cce5630",
+ "85f423e4aff3466981171dbadc44d228",
+ "3be762feea6c4a1e8132e0d76ae7f31e",
+ "6e311a0e8c36447187bc30a098f398dd",
+ "11230091b51e4f47b3ad90512d228d02",
+ "e50b6c7fa38945e78beb1c65f3723c8d",
+ "931f1ddceae642e883959da2c628b182",
+ "826552dc751b425c94bef2cbc1226544",
+ "cb412b2dfe314efd9616ce83f6a82498",
+ "ce08d058513441c4be31d98074642087",
+ "9c8b3a1dfc3647bdbcd7609b877a7d04"
+ ]
+ }
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "Map: 0%| | 0/9033 [00:00[INST] What is the typical wattage of bulb in a lightbox?[/INST] The typical wattage of a bulb in a lightbox is 60 watts, although domestic LED bulbs are normally much lower than 60 watts, as they produce the same or greater lumens for less wattage than alternatives. A 60-watt Equivalent LED bulb can be calculated using the 7:1 ratio, which divides 60 watts by 7 to get roughly 9 watts.[INST] Rewrite your description of the typical wattage of a bulb in a lightbox to only include the key points in a list format.[/INST]\n"
+ ]
+ }
+ ],
+ "source": [
+ "dataset = dataset.map(\n",
+ " lambda examples: {\n",
+ " \"text\": [\n",
+ " tokenizer.apply_chat_template(\n",
+ " messages,\n",
+ " tokenize=False,\n",
+ " add_generation_prompt=False\n",
+ " ) if all(\"role\" in msg and \"content\" in msg for msg in messages) else\n",
+ " (\n",
+ " \"\" +\n",
+ " \"\".join(\n",
+ " f\"[INST] {msg['content']} [/INST]\" if msg[\"role\"] == \"user\" else\n",
+ " f\" {msg['content']}\" if msg[\"role\"] == \"assistant\" else \"\"\n",
+ " for msg in messages\n",
+ " )\n",
+ " )\n",
+ " for messages in examples[\"messages\"]\n",
+ " ]\n",
+ " },\n",
+ " batched=True,\n",
+ " remove_columns=[\"messages\"]\n",
+ ")\n",
+ "\n",
+ "print(dataset[5][\"text\"]) # see one example"
+ ]
},
{
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "f50eabd823c540dbaf4f3dc49364093d",
- "version_major": 2,
- "version_minor": 0
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "idAEIeSQ3xdS"
},
- "text/plain": [
- "Map (num_proc=2): 0%| | 0/9033 [00:00\n",
+ "### Train the model\n",
+ "Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support TRL's `DPOTrainer`!"
]
- },
- "metadata": {},
- "output_type": "display_data"
},
{
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "max_steps is given, it will override any value given in num_train_epochs\n"
- ]
- }
- ],
- "source": [
- "from trl import SFTConfig, SFTTrainer\n",
- "trainer = SFTTrainer(\n",
- " model = model,\n",
- " tokenizer = tokenizer,\n",
- " train_dataset = dataset,\n",
- " dataset_text_field = \"text\",\n",
- " max_seq_length = max_seq_length,\n",
- " packing = False, # Can make training 5x faster for short sequences.\n",
- " args = SFTConfig(\n",
- " per_device_train_batch_size = 2,\n",
- " gradient_accumulation_steps = 4,\n",
- " warmup_steps = 5,\n",
- " max_steps = 60,\n",
- " learning_rate = 2e-4,\n",
- " logging_steps = 1,\n",
- " optim = \"adamw_8bit\",\n",
- " weight_decay = 0.01,\n",
- " lr_scheduler_type = \"linear\",\n",
- " seed = 3407,\n",
- " output_dir = \"outputs\",\n",
- " report_to = \"none\", # Use this for WandB etc\n",
- " ),\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "colab": {
- "base_uri": "https://localhost:8080/"
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 49,
+ "referenced_widgets": [
+ "cd4112b2d10d4793861dd0a91f895354",
+ "7f018f774973423c93c75665395c524f",
+ "7f837996179f42b79ee68470ffe00065",
+ "c9216f2a1a9d40c0bd84403b2a4b48ba",
+ "2e35113e93bd4642b68bf20339c3d8be",
+ "3ecabf86b35841ca9a0a7e8888320ff7",
+ "fa3d4ea0ada74d7ab6f1153adce72a8d",
+ "aebb721c7eb24496945a9b5743ecf1f7",
+ "1fe6e02569784215a8b843fefe8f5c84",
+ "b6a51d5c089c40eda4ee51f5140ade1b",
+ "2a7d7913860e42aeae293cecfc10582a"
+ ]
+ },
+ "id": "95_Nn-89DhsL",
+ "outputId": "351636fe-56e0-4b3d-f8ed-19e5653d7ace"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "Unsloth: Tokenizing [\"text\"] (num_proc=2): 0%| | 0/9033 [00:00"
+ ],
+ "text/html": [
+ "\n",
+ " \n",
+ " \n",
+ "
\n",
+ " [60/60 02:10, Epoch 0/1]\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | Step | \n",
+ " Training Loss | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 1 | \n",
+ " 1.433300 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 1.214400 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 0.993900 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 1.250900 | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " 1.167800 | \n",
+ "
\n",
+ " \n",
+ " | 6 | \n",
+ " 1.058400 | \n",
+ "
\n",
+ " \n",
+ " | 7 | \n",
+ " 1.147400 | \n",
+ "
\n",
+ " \n",
+ " | 8 | \n",
+ " 1.226800 | \n",
+ "
\n",
+ " \n",
+ " | 9 | \n",
+ " 1.023300 | \n",
+ "
\n",
+ " \n",
+ " | 10 | \n",
+ " 1.187600 | \n",
+ "
\n",
+ " \n",
+ " | 11 | \n",
+ " 1.156600 | \n",
+ "
\n",
+ " \n",
+ " | 12 | \n",
+ " 1.003900 | \n",
+ "
\n",
+ " \n",
+ " | 13 | \n",
+ " 1.240700 | \n",
+ "
\n",
+ " \n",
+ " | 14 | \n",
+ " 1.220900 | \n",
+ "
\n",
+ " \n",
+ " | 15 | \n",
+ " 0.905100 | \n",
+ "
\n",
+ " \n",
+ " | 16 | \n",
+ " 0.852800 | \n",
+ "
\n",
+ " \n",
+ " | 17 | \n",
+ " 0.936400 | \n",
+ "
\n",
+ " \n",
+ " | 18 | \n",
+ " 0.882700 | \n",
+ "
\n",
+ " \n",
+ " | 19 | \n",
+ " 0.864500 | \n",
+ "
\n",
+ " \n",
+ " | 20 | \n",
+ " 1.000300 | \n",
+ "
\n",
+ " \n",
+ " | 21 | \n",
+ " 1.010300 | \n",
+ "
\n",
+ " \n",
+ " | 22 | \n",
+ " 0.894100 | \n",
+ "
\n",
+ " \n",
+ " | 23 | \n",
+ " 0.846200 | \n",
+ "
\n",
+ " \n",
+ " | 24 | \n",
+ " 0.708800 | \n",
+ "
\n",
+ " \n",
+ " | 25 | \n",
+ " 0.815600 | \n",
+ "
\n",
+ " \n",
+ " | 26 | \n",
+ " 0.715400 | \n",
+ "
\n",
+ " \n",
+ " | 27 | \n",
+ " 0.402000 | \n",
+ "
\n",
+ " \n",
+ " | 28 | \n",
+ " 0.442000 | \n",
+ "
\n",
+ " \n",
+ " | 29 | \n",
+ " 0.485300 | \n",
+ "
\n",
+ " \n",
+ " | 30 | \n",
+ " 0.497900 | \n",
+ "
\n",
+ " \n",
+ " | 31 | \n",
+ " 0.447300 | \n",
+ "
\n",
+ " \n",
+ " | 32 | \n",
+ " 0.526100 | \n",
+ "
\n",
+ " \n",
+ " | 33 | \n",
+ " 0.319700 | \n",
+ "
\n",
+ " \n",
+ " | 34 | \n",
+ " 0.489800 | \n",
+ "
\n",
+ " \n",
+ " | 35 | \n",
+ " 0.377100 | \n",
+ "
\n",
+ " \n",
+ " | 36 | \n",
+ " 0.226100 | \n",
+ "
\n",
+ " \n",
+ " | 37 | \n",
+ " 0.360400 | \n",
+ "
\n",
+ " \n",
+ " | 38 | \n",
+ " 0.215400 | \n",
+ "
\n",
+ " \n",
+ " | 39 | \n",
+ " 0.276300 | \n",
+ "
\n",
+ " \n",
+ " | 40 | \n",
+ " 0.276000 | \n",
+ "
\n",
+ " \n",
+ " | 41 | \n",
+ " 0.242800 | \n",
+ "
\n",
+ " \n",
+ " | 42 | \n",
+ " 0.236800 | \n",
+ "
\n",
+ " \n",
+ " | 43 | \n",
+ " 0.247800 | \n",
+ "
\n",
+ " \n",
+ " | 44 | \n",
+ " 0.296500 | \n",
+ "
\n",
+ " \n",
+ " | 45 | \n",
+ " 0.278900 | \n",
+ "
\n",
+ " \n",
+ " | 46 | \n",
+ " 0.210000 | \n",
+ "
\n",
+ " \n",
+ " | 47 | \n",
+ " 0.239900 | \n",
+ "
\n",
+ " \n",
+ " | 48 | \n",
+ " 0.254900 | \n",
+ "
\n",
+ " \n",
+ " | 49 | \n",
+ " 0.357900 | \n",
+ "
\n",
+ " \n",
+ " | 50 | \n",
+ " 0.172600 | \n",
+ "
\n",
+ " \n",
+ " | 51 | \n",
+ " 0.286600 | \n",
+ "
\n",
+ " \n",
+ " | 52 | \n",
+ " 0.199400 | \n",
+ "
\n",
+ " \n",
+ " | 53 | \n",
+ " 0.174300 | \n",
+ "
\n",
+ " \n",
+ " | 54 | \n",
+ " 0.325800 | \n",
+ "
\n",
+ " \n",
+ " | 55 | \n",
+ " 0.172700 | \n",
+ "
\n",
+ " \n",
+ " | 56 | \n",
+ " 0.137000 | \n",
+ "
\n",
+ " \n",
+ " | 57 | \n",
+ " 0.149000 | \n",
+ "
\n",
+ " \n",
+ " | 58 | \n",
+ " 0.161900 | \n",
+ "
\n",
+ " \n",
+ " | 59 | \n",
+ " 0.220300 | \n",
+ "
\n",
+ " \n",
+ " | 60 | \n",
+ " 0.203900 | \n",
+ "
\n",
+ " \n",
+ "
"
+ ]
+ },
+ "metadata": {}
+ }
+ ],
+ "source": [
+ "trainer_stats = trainer.train()"
+ ]
},
{
- "data": {
- "text/html": [
- "\n",
- "
\n",
- " \n",
- "
\n",
- " [60/60 14:06, Epoch 0/1]\n",
- "
\n",
- " \n",
- " \n",
- " | Step | \n",
- " Training Loss | \n",
- "
\n",
- " \n",
- " \n",
- " \n",
- " | 1 | \n",
- " 1.645400 | \n",
- "
\n",
- " \n",
- " | 2 | \n",
- " 1.203700 | \n",
- "
\n",
- " \n",
- " | 3 | \n",
- " 1.153700 | \n",
- "
\n",
- " \n",
- " | 4 | \n",
- " 1.208300 | \n",
- "
\n",
- " \n",
- " | 5 | \n",
- " 1.135200 | \n",
- "
\n",
- " \n",
- " | 6 | \n",
- " 1.074600 | \n",
- "
\n",
- " \n",
- " | 7 | \n",
- " 0.829600 | \n",
- "
\n",
- " \n",
- " | 8 | \n",
- " 1.294600 | \n",
- "
\n",
- " \n",
- " | 9 | \n",
- " 1.255400 | \n",
- "
\n",
- " \n",
- " | 10 | \n",
- " 1.066200 | \n",
- "
\n",
- " \n",
- " | 11 | \n",
- " 1.068100 | \n",
- "
\n",
- " \n",
- " | 12 | \n",
- " 0.992700 | \n",
- "
\n",
- " \n",
- " | 13 | \n",
- " 1.228000 | \n",
- "
\n",
- " \n",
- " | 14 | \n",
- " 1.141200 | \n",
- "
\n",
- " \n",
- " | 15 | \n",
- " 0.970500 | \n",
- "
\n",
- " \n",
- " | 16 | \n",
- " 0.934000 | \n",
- "
\n",
- " \n",
- " | 17 | \n",
- " 0.782800 | \n",
- "
\n",
- " \n",
- " | 18 | \n",
- " 0.852000 | \n",
- "
\n",
- " \n",
- " | 19 | \n",
- " 0.915100 | \n",
- "
\n",
- " \n",
- " | 20 | \n",
- " 1.198800 | \n",
- "
\n",
- " \n",
- " | 21 | \n",
- " 0.960000 | \n",
- "
\n",
- " \n",
- " | 22 | \n",
- " 0.893900 | \n",
- "
\n",
- " \n",
- " | 23 | \n",
- " 1.001500 | \n",
- "
\n",
- " \n",
- " | 24 | \n",
- " 0.775700 | \n",
- "
\n",
- " \n",
- " | 25 | \n",
- " 0.979500 | \n",
- "
\n",
- " \n",
- " | 26 | \n",
- " 1.042700 | \n",
- "
\n",
- " \n",
- " | 27 | \n",
- " 0.842900 | \n",
- "
\n",
- " \n",
- " | 28 | \n",
- " 0.967600 | \n",
- "
\n",
- " \n",
- " | 29 | \n",
- " 1.057900 | \n",
- "
\n",
- " \n",
- " | 30 | \n",
- " 0.948900 | \n",
- "
\n",
- " \n",
- " | 31 | \n",
- " 1.137300 | \n",
- "
\n",
- " \n",
- " | 32 | \n",
- " 1.179300 | \n",
- "
\n",
- " \n",
- " | 33 | \n",
- " 0.822800 | \n",
- "
\n",
- " \n",
- " | 34 | \n",
- " 1.219400 | \n",
- "
\n",
- " \n",
- " | 35 | \n",
- " 1.176800 | \n",
- "
\n",
- " \n",
- " | 36 | \n",
- " 0.995800 | \n",
- "
\n",
- " \n",
- " | 37 | \n",
- " 0.818200 | \n",
- "
\n",
- " \n",
- " | 38 | \n",
- " 0.965200 | \n",
- "
\n",
- " \n",
- " | 39 | \n",
- " 1.107700 | \n",
- "
\n",
- " \n",
- " | 40 | \n",
- " 1.099300 | \n",
- "
\n",
- " \n",
- " | 41 | \n",
- " 0.913600 | \n",
- "
\n",
- " \n",
- " | 42 | \n",
- " 0.998300 | \n",
- "
\n",
- " \n",
- " | 43 | \n",
- " 1.154400 | \n",
- "
\n",
- " \n",
- " | 44 | \n",
- " 1.044700 | \n",
- "
\n",
- " \n",
- " | 45 | \n",
- " 1.002000 | \n",
- "
\n",
- " \n",
- " | 46 | \n",
- " 1.114100 | \n",
- "
\n",
- " \n",
- " | 47 | \n",
- " 1.063500 | \n",
- "
\n",
- " \n",
- " | 48 | \n",
- " 1.052900 | \n",
- "
\n",
- " \n",
- " | 49 | \n",
- " 1.470200 | \n",
- "
\n",
- " \n",
- " | 50 | \n",
- " 1.110300 | \n",
- "
\n",
- " \n",
- " | 51 | \n",
- " 1.243000 | \n",
- "
\n",
- " \n",
- " | 52 | \n",
- " 1.116600 | \n",
- "
\n",
- " \n",
- " | 53 | \n",
- " 0.884500 | \n",
- "
\n",
- " \n",
- " | 54 | \n",
- " 1.116200 | \n",
- "
\n",
- " \n",
- " | 55 | \n",
- " 0.793500 | \n",
- "
\n",
- " \n",
- " | 56 | \n",
- " 1.202300 | \n",
- "
\n",
- " \n",
- " | 57 | \n",
- " 1.093400 | \n",
- "
\n",
- " \n",
- " | 58 | \n",
- " 1.471100 | \n",
- "
\n",
- " \n",
- " | 59 | \n",
- " 1.014300 | \n",
- "
\n",
- " \n",
- " | 60 | \n",
- " 0.828700 | \n",
- "
\n",
- " \n",
- "
"
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "pCqnaKmlO1U9",
+ "outputId": "e76d053b-e958-4c2d-b7ca-7f4a7111204c"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "870.2993 seconds used for training.\n",
+ "14.5 minutes used for training.\n",
+ "Peak reserved memory = 6.289 GB.\n",
+ "Peak reserved memory for training = 1.769 GB.\n",
+ "Peak reserved memory % of max memory = 42.643 %.\n",
+ "Peak reserved memory for training % of max memory = 11.995 %.\n"
+ ]
+ }
],
- "text/plain": [
- ""
+ "source": [
+ "# @title Show final memory and time stats\n",
+ "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
+ "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
+ "used_percentage = round(used_memory / max_memory * 100, 3)\n",
+ "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
+ "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
+ "print(\n",
+ " f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
+ ")\n",
+ "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
+ "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
+ "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
+ "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "trainer_stats = trainer.train()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "colab": {
- "base_uri": "https://localhost:8080/"
},
- "id": "pCqnaKmlO1U9",
- "outputId": "e76d053b-e958-4c2d-b7ca-7f4a7111204c"
- },
- "outputs": [
{
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "870.2993 seconds used for training.\n",
- "14.5 minutes used for training.\n",
- "Peak reserved memory = 6.289 GB.\n",
- "Peak reserved memory for training = 1.769 GB.\n",
- "Peak reserved memory % of max memory = 42.643 %.\n",
- "Peak reserved memory for training % of max memory = 11.995 %.\n"
- ]
- }
- ],
- "source": [
- "# @title Show final memory and time stats\n",
- "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
- "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
- "used_percentage = round(used_memory / max_memory * 100, 3)\n",
- "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
- "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
- "print(\n",
- " f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
- ")\n",
- "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
- "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
- "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
- "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ekOmTR1hSNcr"
- },
- "source": [
- "\n",
- "### Inference\n",
- "Let's run the model! Since we're using `ChatML`, use `apply_chat_template` with `add_generation_prompt` set to `True` for inference."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ekOmTR1hSNcr"
+ },
+ "source": [
+ "\n",
+ "### Inference\n",
+ "Let's run the model! Since we're using `ChatML`, use `apply_chat_template` with `add_generation_prompt` set to `True` for inference."
+ ]
},
- "id": "kR3gIAX-SM2q",
- "outputId": "aeb38973-e654-41cc-a989-c1db359b8489"
- },
- "outputs": [
{
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Unsloth: Will map <|im_end|> to EOS = <|im_end|>.\n",
- "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
- "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
- ]
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "kR3gIAX-SM2q",
+ "outputId": "889bed98-c55e-4a8f-f88c-0fa5620443cc"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Input: [INST] Continue the fibonacci sequence: 1, 1, 2, 3, 5, 8,[/INST]\n",
+ "Response: Here are the next ten numbers in the Fibonacci sequence:\n",
+ "\n",
+ "13, 21, 34, 55, 89, 144, 233, 377, 610, 987.\n"
+ ]
+ }
+ ],
+ "source": [
+ "FastLanguageModel.for_inference(model)\n",
+ "\n",
+ "messages = [{\"role\": \"user\", \"content\": \"Continue the fibonacci sequence: 1, 1, 2, 3, 5, 8,\"}]\n",
+ "inputs = tokenizer.apply_chat_template(\n",
+ " messages,\n",
+ " tokenize=True,\n",
+ " add_generation_prompt=True,\n",
+ " return_tensors=\"pt\"\n",
+ ").to(\"cuda\")\n",
+ "\n",
+ "print(f\"Input: {tokenizer.decode(inputs[0], skip_special_tokens=False)}\")\n",
+ "\n",
+ "outputs = model.generate(\n",
+ " input_ids=inputs,\n",
+ " max_new_tokens=64,\n",
+ " do_sample=True,\n",
+ " temperature=0.7,\n",
+ " top_p=0.9,\n",
+ " repetition_penalty=1.1,\n",
+ " pad_token_id=tokenizer.eos_token_id,\n",
+ " eos_token_id=tokenizer.eos_token_id,\n",
+ ")\n",
+ "\n",
+ "response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)\n",
+ "print(f\"Response: {response}\")"
+ ]
},
{
- "data": {
- "text/plain": [
- "['<|im_start|>user\\nContinue the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|> \\n<|im_start|>assistant\\nThe next number in the Fibonacci sequence is 13.\\n\\nThe Fibonacci sequence is a series of numbers in which each number is the sum of the two preceding ones, starting from 0 and 1. The sequence goes: 0, 1, 1, ']"
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "CrSvZObor0lY"
+ },
+ "source": [
+ " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"
]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from unsloth.chat_templates import get_chat_template\n",
- "\n",
- "tokenizer = get_chat_template(\n",
- " tokenizer,\n",
- " chat_template = \"chatml\", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth\n",
- " mapping = {\"role\" : \"from\", \"content\" : \"value\", \"user\" : \"human\", \"assistant\" : \"gpt\"}, # ShareGPT style\n",
- " map_eos_token = True, # Maps <|im_end|> to instead\n",
- ")\n",
- "\n",
- "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
- "\n",
- "messages = [\n",
- " {\"from\": \"human\", \"value\": \"Continue the fibonnaci sequence: 1, 1, 2, 3, 5, 8,\"},\n",
- "]\n",
- "inputs = tokenizer.apply_chat_template(\n",
- " messages,\n",
- " tokenize = True,\n",
- " add_generation_prompt = True, # Must add for generation\n",
- " return_tensors = \"pt\",\n",
- ").to(\"cuda\")\n",
- "\n",
- "outputs = model.generate(input_ids = inputs, max_new_tokens = 64, use_cache = True)\n",
- "tokenizer.batch_decode(outputs)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "CrSvZObor0lY"
- },
- "source": [
- " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
},
- "id": "e2pEuRb1r2Vg",
- "outputId": "5163ef93-7952-43fb-fe01-7dea089406a3"
- },
- "outputs": [
{
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
- "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
- ]
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "e2pEuRb1r2Vg",
+ "outputId": "bf6096c3-8138-4b7d-9f16-519e78f00afe"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Input: [INST] Continue the fibonacci sequence: 1, 1, 2, 3, 5, 8,[/INST]\n",
+ "The next number in the Fibonacci sequence is 13.\n",
+ "\n",
+ "To find the next number, you simply add the previous two numbers together: 8 + 5 = 13.\n",
+ "\n",
+ "This pattern continues for every subsequent number in the sequence. The next few numbers after 13 are\n"
+ ]
+ }
+ ],
+ "source": [
+ "FastLanguageModel.for_inference(model)\n",
+ "\n",
+ "messages = [{\"role\": \"user\", \"content\": \"Continue the fibonacci sequence: 1, 1, 2, 3, 5, 8,\"}]\n",
+ "inputs = tokenizer.apply_chat_template(\n",
+ " messages,\n",
+ " tokenize=True,\n",
+ " add_generation_prompt=True,\n",
+ " return_tensors=\"pt\"\n",
+ ").to(\"cuda\")\n",
+ "\n",
+ "print(f\"Input: {tokenizer.decode(inputs[0], skip_special_tokens=False)}\")\n",
+ "\n",
+ "from transformers import TextStreamer\n",
+ "\n",
+ "text_streamer = TextStreamer(\n",
+ " tokenizer,\n",
+ " skip_prompt=True,\n",
+ " skip_special_tokens=True,\n",
+ ")\n",
+ "\n",
+ "outputs = model.generate(\n",
+ " input_ids=inputs,\n",
+ " streamer=text_streamer,\n",
+ " max_new_tokens=64,\n",
+ " do_sample=True,\n",
+ " temperature=0.7,\n",
+ " top_p=0.9,\n",
+ " repetition_penalty=1.1,\n",
+ " pad_token_id=tokenizer.eos_token_id,\n",
+ " eos_token_id=tokenizer.eos_token_id,\n",
+ ")"
+ ]
},
{
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "<|im_start|>user\n",
- "Continue the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|> \n",
- "<|im_start|>assistant\n",
- "The next number in the Fibonacci sequence is 13.\n",
- "\n",
- "The Fibonacci sequence is a series of numbers in which each number is the sum of the two preceding ones, starting from 0 and 1. The sequence goes: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2\n"
- ]
- }
- ],
- "source": [
- "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
- "\n",
- "messages = [\n",
- " {\"from\": \"human\", \"value\": \"Continue the fibonnaci sequence: 1, 1, 2, 3, 5, 8,\"},\n",
- "]\n",
- "inputs = tokenizer.apply_chat_template(\n",
- " messages,\n",
- " tokenize = True,\n",
- " add_generation_prompt = True, # Must add for generation\n",
- " return_tensors = \"pt\",\n",
- ").to(\"cuda\")\n",
- "\n",
- "from transformers import TextStreamer\n",
- "text_streamer = TextStreamer(tokenizer)\n",
- "_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "uMuVrWbjAzhc"
- },
- "source": [
- "\n",
- "### Saving, loading finetuned models\n",
- "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
- "\n",
- "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "uMuVrWbjAzhc"
+ },
+ "source": [
+ "\n",
+ "### Saving, loading finetuned models\n",
+ "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
+ "\n",
+ "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
+ ]
},
- "id": "upcOlWe7A1vc",
- "outputId": "e58bbc7d-41c5-47f4-c780-df1fd36f1d96"
- },
- "outputs": [
{
- "data": {
- "text/plain": [
- "('lora_model/tokenizer_config.json',\n",
- " 'lora_model/special_tokens_map.json',\n",
- " 'lora_model/tokenizer.json')"
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "upcOlWe7A1vc",
+ "outputId": "621c22de-f8df-4fb3-8649-0e5f1971a20f"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "('lora_model/tokenizer_config.json',\n",
+ " 'lora_model/special_tokens_map.json',\n",
+ " 'lora_model/chat_template.jinja',\n",
+ " 'lora_model/tokenizer.model',\n",
+ " 'lora_model/added_tokens.json',\n",
+ " 'lora_model/tokenizer.json')"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 81
+ }
+ ],
+ "source": [
+ "model.save_pretrained(\"lora_model\") # Local saving\n",
+ "tokenizer.save_pretrained(\"lora_model\")\n",
+ "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n",
+ "# tokenizer.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving"
]
- },
- "execution_count": 14,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "model.save_pretrained(\"lora_model\") # Local saving\n",
- "tokenizer.save_pretrained(\"lora_model\")\n",
- "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n",
- "# tokenizer.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "AEEcJ4qfC7Lp"
- },
- "source": [
- "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
},
- "id": "MKX_XKs_BNZR",
- "outputId": "34318835-3534-48a7-a5c5-f8a2a12705b9"
- },
- "outputs": [
{
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
- "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
- ]
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "AEEcJ4qfC7Lp"
+ },
+ "source": [
+ "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
+ ]
},
{
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "<|im_start|>user\n",
- "What is a famous tall tower in Paris?<|im_end|> \n",
- "<|im_start|>assistant\n",
- "The Eiffel Tower is a famous tall tower in Paris. It is one of the most recognizable landmarks in the world and is a popular tourist destination. The tower was built in 1889 for the World's Fair and is named after Gustave Eiffel, the engineer who designed and built it. The tower stands at a height of 324 meters (1,063 feet) and is made of iron. It is located on the Champ de Mars, a large public park in Paris.<|im_end|>\n"
- ]
- }
- ],
- "source": [
- "if False:\n",
- " from unsloth import FastLanguageModel\n",
- " model, tokenizer = FastLanguageModel.from_pretrained(\n",
- " model_name = \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
- " max_seq_length = max_seq_length,\n",
- " dtype = dtype,\n",
- " load_in_4bit = load_in_4bit,\n",
- " )\n",
- " FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
- "\n",
- "messages = [\n",
- " {\"from\": \"human\", \"value\": \"What is a famous tall tower in Paris?\"},\n",
- "]\n",
- "inputs = tokenizer.apply_chat_template(\n",
- " messages,\n",
- " tokenize = True,\n",
- " add_generation_prompt = True, # Must add for generation\n",
- " return_tensors = \"pt\",\n",
- ").to(\"cuda\")\n",
- "\n",
- "from transformers import TextStreamer\n",
- "text_streamer = TextStreamer(tokenizer)\n",
- "_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "QQMjaNrjsU5_"
- },
- "source": [
- "You can also use Hugging Face's `AutoModelForPeftCausalLM`. Only use this if you do not have `unsloth` installed. It can be hopelessly slow, since `4bit` model downloading is not supported, and Unsloth's **inference is 2x faster**."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "yFfaXG0WsQuE"
- },
- "outputs": [],
- "source": [
- "if False:\n",
- " # I highly do NOT suggest - use Unsloth if possible\n",
- " from peft import AutoModelForPeftCausalLM\n",
- " from transformers import AutoTokenizer\n",
- "\n",
- " model = AutoModelForPeftCausalLM.from_pretrained(\n",
- " \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
- " load_in_4bit=load_in_4bit,\n",
- " )\n",
- " tokenizer = AutoTokenizer.from_pretrained(\"lora_model\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "f422JgM9sdVT"
- },
- "source": [
- "### Saving to float16 for VLLM\n",
- "\n",
- "We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "iHjt_SMYsd3P"
- },
- "outputs": [],
- "source": [
- "# Merge to 16bit\n",
- "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_16bit\",)\n",
- "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_16bit\", token = \"\")\n",
- "\n",
- "# Merge to 4bit\n",
- "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_4bit\",)\n",
- "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_4bit\", token = \"\")\n",
- "\n",
- "# Just LoRA adapters\n",
- "if False:\n",
- " model.save_pretrained(\"model\")\n",
- " tokenizer.save_pretrained(\"model\")\n",
- "if False:\n",
- " model.push_to_hub(\"hf/model\", token = \"\")\n",
- " tokenizer.push_to_hub(\"hf/model\", token = \"\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "TCv4vXHd61i7"
- },
- "source": [
- "### GGUF / llama.cpp Conversion\n",
- "To save to `GGUF` / `llama.cpp`, we support it natively now! We clone `llama.cpp` and we default save it to `q8_0`. We allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.\n",
- "\n",
- "Some supported quant methods (full list on our [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):\n",
- "* `q8_0` - Fast conversion. High resource use, but generally acceptable.\n",
- "* `q4_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.\n",
- "* `q5_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "FqfebeAdT073"
- },
- "outputs": [],
- "source": [
- "# Save to 8bit Q8_0\n",
- "if False: model.save_pretrained_gguf(\"model\", tokenizer,)\n",
- "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, token = \"\")\n",
- "\n",
- "# Save to 16bit GGUF\n",
- "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"f16\")\n",
- "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"f16\", token = \"\")\n",
- "\n",
- "# Save to q4_k_m GGUF\n",
- "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"q4_k_m\")\n",
- "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"q4_k_m\", token = \"\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now, use the `model-unsloth.gguf` file or `model-unsloth-Q4_K_M.gguf` file in llama.cpp or a UI based system like Jan or Open WebUI. You can install Jan [here](https://github.com/janhq/jan) and Open WebUI [here](https://github.com/open-webui/open-webui)\n",
- "\n",
- "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
- "\n",
- "Some other links:\n",
- "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n",
- "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n",
- "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n",
- "6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n",
- "\n",
- "\n",
- "
![](\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\")
\n",
- "
![](\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\")
\n",
- "
![](\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\")
\n",
- "\n",
- " Join Discord if you need help + \u2b50\ufe0f
Star us on Github \u2b50\ufe0f\n",
- "
\n",
- "
\n",
- "
![](\"https://github.com/unslothai/unsloth/raw/main/images/Discord)
\n",
- "
Join Discord if you need help + \u2b50
Star us on Github \u2b50\n",
- "
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
"
+ ]
},
- "id": "k_R0oWSwi7JV",
- "outputId": "224dee66-de62-49a3-d821-a2cb87936a46"
- },
- "outputs": [
{
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "==((====))== Unsloth 2025.5.4: Fast Csm patching. Transformers: 4.52.0.dev0.\n",
- " \\\\ /| Tesla T4. Num GPUs = 1. Max memory: 14.741 GB. Platform: Linux.\n",
- "O^O/ \\_/ \\ Torch: 2.6.0+cu124. CUDA: 7.5. CUDA Toolkit: 12.4. Triton: 3.2.0\n",
- "\\ / Bfloat16 = FALSE. FA [Xformers = 0.0.29.post3. FA2 = False]\n",
- " \"-____-\" Free license: http://github.com/unslothai/unsloth\n",
- "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n",
- "Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA.\n",
- "unsloth/csm-1b does not have a padding token! Will use pad_token = <|PAD_TOKEN|>.\n"
- ]
- }
- ],
- "source": [
- "from unsloth import FastModel\n",
- "from transformers import CsmForConditionalGeneration\n",
- "import torch\n",
- "\n",
- "model, processor = FastModel.from_pretrained(\n",
- " model_name = \"unsloth/csm-1b\",\n",
- " max_seq_length= 2048, # Choose any for long context!\n",
- " dtype = None, # Leave as None for auto-detection\n",
- " auto_model = CsmForConditionalGeneration,\n",
- " load_in_4bit = False, # Select True for 4bit - reduces memory usage\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "DPTVeI9ni7JY"
- },
- "source": [
- "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hZ4haS5Bihlt"
+ },
+ "source": [
+ "To run this, press \"*Runtime*\" and press \"*Run all*\" on a **free** Tesla T4 Google Colab instance!\n",
+ "\n",
+ "
\n",
+ "
\n",
+ "
Join Discord if you need help + ⭐
Star us on Github ⭐\n",
+ "
\n",
- " \n",
- "
\n",
- " [60/60 04:10, Epoch 1/2]\n",
- "
\n",
- " \n",
- " \n",
- " | Step | \n",
- " Training Loss | \n",
- "
\n",
- " \n",
- " \n",
- " \n",
- " | 1 | \n",
- " 4.807000 | \n",
- "
\n",
- " \n",
- " | 2 | \n",
- " 5.321600 | \n",
- "
\n",
- " \n",
- " | 3 | \n",
- " 5.008500 | \n",
- "
\n",
- " \n",
- " | 4 | \n",
- " 4.953200 | \n",
- "
\n",
- " \n",
- " | 5 | \n",
- " 5.235600 | \n",
- "
\n",
- " \n",
- " | 6 | \n",
- " 4.338500 | \n",
- "
\n",
- " \n",
- " | 7 | \n",
- " 4.611100 | \n",
- "
\n",
- " \n",
- " | 8 | \n",
- " 4.968200 | \n",
- "
\n",
- " \n",
- " | 9 | \n",
- " 5.087200 | \n",
- "
\n",
- " \n",
- " | 10 | \n",
- " 5.169500 | \n",
- "
\n",
- " \n",
- " | 11 | \n",
- " 4.770600 | \n",
- "
\n",
- " \n",
- " | 12 | \n",
- " 4.939600 | \n",
- "
\n",
- " \n",
- " | 13 | \n",
- " 4.730200 | \n",
- "
\n",
- " \n",
- " | 14 | \n",
- " 5.205600 | \n",
- "
\n",
- " \n",
- " | 15 | \n",
- " 5.272200 | \n",
- "
\n",
- " \n",
- " | 16 | \n",
- " 5.296400 | \n",
- "
\n",
- " \n",
- " | 17 | \n",
- " 4.949700 | \n",
- "
\n",
- " \n",
- " | 18 | \n",
- " 5.393700 | \n",
- "
\n",
- " \n",
- " | 19 | \n",
- " 5.344200 | \n",
- "
\n",
- " \n",
- " | 20 | \n",
- " 5.344900 | \n",
- "
\n",
- " \n",
- " | 21 | \n",
- " 4.878800 | \n",
- "
\n",
- " \n",
- " | 22 | \n",
- " 5.259400 | \n",
- "
\n",
- " \n",
- " | 23 | \n",
- " 4.935000 | \n",
- "
\n",
- " \n",
- " | 24 | \n",
- " 4.803700 | \n",
- "
\n",
- " \n",
- " | 25 | \n",
- " 5.222400 | \n",
- "
\n",
- " \n",
- " | 26 | \n",
- " 5.021800 | \n",
- "
\n",
- " \n",
- " | 27 | \n",
- " 5.427000 | \n",
- "
\n",
- " \n",
- " | 28 | \n",
- " 5.058500 | \n",
- "
\n",
- " \n",
- " | 29 | \n",
- " 4.970100 | \n",
- "
\n",
- " \n",
- " | 30 | \n",
- " 5.093200 | \n",
- "
\n",
- " \n",
- " | 31 | \n",
- " 4.951000 | \n",
- "
\n",
- " \n",
- " | 32 | \n",
- " 5.057200 | \n",
- "
\n",
- " \n",
- " | 33 | \n",
- " 5.226800 | \n",
- "
\n",
- " \n",
- " | 34 | \n",
- " 5.330500 | \n",
- "
\n",
- " \n",
- " | 35 | \n",
- " 4.907200 | \n",
- "
\n",
- " \n",
- " | 36 | \n",
- " 5.319300 | \n",
- "
\n",
- " \n",
- " | 37 | \n",
- " 4.526100 | \n",
- "
\n",
- " \n",
- " | 38 | \n",
- " 4.460400 | \n",
- "
\n",
- " \n",
- " | 39 | \n",
- " 5.393900 | \n",
- "
\n",
- " \n",
- " | 40 | \n",
- " 5.403600 | \n",
- "
\n",
- " \n",
- " | 41 | \n",
- " 4.773700 | \n",
- "
\n",
- " \n",
- " | 42 | \n",
- " 4.352500 | \n",
- "
\n",
- " \n",
- " | 43 | \n",
- " 5.234400 | \n",
- "
\n",
- " \n",
- " | 44 | \n",
- " 4.750500 | \n",
- "
\n",
- " \n",
- " | 45 | \n",
- " 4.649300 | \n",
- "
\n",
- " \n",
- " | 46 | \n",
- " 5.191500 | \n",
- "
\n",
- " \n",
- " | 47 | \n",
- " 4.782700 | \n",
- "
\n",
- " \n",
- " | 48 | \n",
- " 5.004000 | \n",
- "
\n",
- " \n",
- " | 49 | \n",
- " 4.774600 | \n",
- "
\n",
- " \n",
- " | 50 | \n",
- " 5.218100 | \n",
- "
\n",
- " \n",
- " | 51 | \n",
- " 4.785000 | \n",
- "
\n",
- " \n",
- " | 52 | \n",
- " 4.747500 | \n",
- "
\n",
- " \n",
- " | 53 | \n",
- " 4.617100 | \n",
- "
\n",
- " \n",
- " | 54 | \n",
- " 5.153500 | \n",
- "
\n",
- " \n",
- " | 55 | \n",
- " 4.475000 | \n",
- "
\n",
- " \n",
- " | 56 | \n",
- " 4.918500 | \n",
- "
\n",
- " \n",
- " | 57 | \n",
- " 5.189200 | \n",
- "
\n",
- " \n",
- " | 58 | \n",
- " 4.105400 | \n",
- "
\n",
- " \n",
- " | 59 | \n",
- " 4.668900 | \n",
- "
\n",
- " \n",
- " | 60 | \n",
- " 4.730600 | \n",
- "
\n",
- " \n",
- "
"
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "A-LqUgufi7JU"
+ },
+ "source": [
+ "### Unsloth\n",
+ "\n",
+ "`FastModel` supports loading nearly any model now! This includes Vision and Text models!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "k_R0oWSwi7JV",
+ "outputId": "0c0e176b-2a3c-46b1-cf9f-2f836f8f36d6"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.\n",
+ "🦥 Unsloth Zoo will now patch everything to make training faster!\n",
+ "==((====))== Unsloth 2025.9.7: Fast Csm patching. Transformers: 4.52.3.\n",
+ " \\\\ /| Tesla T4. Num GPUs = 1. Max memory: 14.741 GB. Platform: Linux.\n",
+ "O^O/ \\_/ \\ Torch: 2.8.0+cu126. CUDA: 7.5. CUDA Toolkit: 12.6. Triton: 3.4.0\n",
+ "\\ / Bfloat16 = FALSE. FA [Xformers = 0.0.32.post2. FA2 = False]\n",
+ " \"-____-\" Free license: http://github.com/unslothai/unsloth\n",
+ "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n",
+ "Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA.\n",
+ "unsloth/csm-1b does not have a padding token! Will use pad_token = <|PAD_TOKEN|>.\n"
+ ]
+ }
],
- "text/plain": [
- ""
+ "source": [
+ "from unsloth import FastModel\n",
+ "from transformers import CsmForConditionalGeneration\n",
+ "import torch\n",
+ "\n",
+ "model, processor = FastModel.from_pretrained(\n",
+ " model_name = \"unsloth/csm-1b\",\n",
+ " max_seq_length= 2048, # Choose any for long context!\n",
+ " dtype = None, # Leave as None for auto-detection\n",
+ " auto_model = CsmForConditionalGeneration,\n",
+ " load_in_4bit = False, # Select True for 4bit - reduces memory usage\n",
+ ")"
]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "trainer_stats = trainer.train()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "cellView": "form",
- "colab": {
- "base_uri": "https://localhost:8080/"
},
- "id": "SjWdtI8zi7Jv",
- "outputId": "f08ea38e-de70-47bc-f517-61ee00f18c6b"
- },
- "outputs": [
{
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "313.8963 seconds used for training.\n",
- "5.23 minutes used for training.\n",
- "Peak reserved memory = 6.719 GB.\n",
- "Peak reserved memory for training = 0.0 GB.\n",
- "Peak reserved memory % of max memory = 45.58 %.\n",
- "Peak reserved memory for training % of max memory = 0.0 %.\n"
- ]
- }
- ],
- "source": [
- "# @title Show final memory and time stats\n",
- "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
- "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
- "used_percentage = round(used_memory / max_memory * 100, 3)\n",
- "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
- "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
- "print(\n",
- " f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
- ")\n",
- "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
- "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
- "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
- "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "JWCCkIXyi7Jy"
- },
- "source": [
- "\n",
- "### Inference\n",
- "Let's run the model! You can change the prompts"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "DPTVeI9ni7JY"
+ },
+ "source": [
+ "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"
+ ]
+ },
{
- "data": {
- "text/html": [
- "\n",
- " \n",
- " "
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "_dk5azmwi7Jb",
+ "outputId": "13c07348-a3ba-477a-9439-e5355ebc48df"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Unsloth: Making `model.base_model.model.backbone_model` require gradients\n"
+ ]
+ }
],
- "text/plain": [
- ""
+ "source": [
+ "model = FastModel.get_peft_model(\n",
+ " model,\n",
+ " r = 32, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",
+ " target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
+ " \"gate_proj\", \"up_proj\", \"down_proj\",],\n",
+ " lora_alpha = 32,\n",
+ " lora_dropout = 0, # Supports any, but = 0 is optimized\n",
+ " bias = \"none\", # Supports any, but = \"none\" is optimized\n",
+ " # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",
+ " use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n",
+ " random_state = 3407,\n",
+ " use_rslora = False, # We support rank stabilized LoRA\n",
+ " loftq_config = None, # And LoftQ\n",
+ ")"
]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "from IPython.display import Audio, display\n",
- "import soundfile as sf\n",
- "\n",
- "text = \"We just finished fine tuning a text to speech model... and it's pretty good!\"\n",
- "speaker_id = 0\n",
- "inputs = processor(f\"[{speaker_id}]{text}\", add_special_tokens=True).to(\"cuda\")\n",
- "audio_values = model.generate(\n",
- " **inputs,\n",
- " max_new_tokens=125, # 125 tokens is 10 seconds of audio, for longer speech increase this\n",
- " # play with these parameters to tweak results\n",
- " # depth_decoder_top_k=0,\n",
- " # depth_decoder_top_p=0.9,\n",
- " # depth_decoder_do_sample=True,\n",
- " # depth_decoder_temperature=0.9,\n",
- " # top_k=0,\n",
- " # top_p=1.0,\n",
- " # temperature=0.9,\n",
- " # do_sample=True,\n",
- " #########################################################\n",
- " output_audio=True\n",
- ")\n",
- "audio = audio_values[0].to(torch.float32).cpu().numpy()\n",
- "sf.write(\"example_without_context.wav\", audio, 24000)\n",
- "display(Audio(audio, rate=24000))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [
+ },
{
- "data": {
- "text/html": [
- "\n",
- " \n",
- " "
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cflyBsb9i7Je"
+ },
+ "source": [
+ "\n",
+ "### Data Prep \n",
+ "\n",
+ "We will use the `MrDragonFox/Elise`, which is designed for training TTS models. Ensure that your dataset follows the required format: **text, audio** for single-speaker models or **source, text, audio** for multi-speaker models. You can modify this section to accommodate your own dataset, but maintaining the correct structure is essential for optimal training."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "cellView": "form",
+ "id": "QeCRvPI5i7Jh",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "b8884dd2-0ed9-46df-9362-80a58cae155e"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Unsloth: No speaker found, adding default \"source\" of 0 for all examples\n"
+ ]
+ }
],
- "text/plain": [
- ""
+ "source": [
+ "#@title Dataset Prep functions\n",
+ "from datasets import load_dataset, Audio, Dataset\n",
+ "import os\n",
+ "from transformers import AutoProcessor\n",
+ "processor = AutoProcessor.from_pretrained(\"unsloth/csm-1b\")\n",
+ "\n",
+ "raw_ds = load_dataset(\"MrDragonFox/Elise\", split=\"train\")\n",
+ "\n",
+ "# Getting the speaker id is important for multi-speaker models and speaker consistency\n",
+ "speaker_key = \"source\"\n",
+ "if \"source\" not in raw_ds.column_names and \"speaker_id\" not in raw_ds.column_names:\n",
+ " print(\"Unsloth: No speaker found, adding default \\\"source\\\" of 0 for all examples\")\n",
+ " new_column = [\"0\"] * len(raw_ds)\n",
+ " raw_ds = raw_ds.add_column(\"source\", new_column)\n",
+ "elif \"source\" not in raw_ds.column_names and \"speaker_id\" in raw_ds.column_names:\n",
+ " speaker_key = \"speaker_id\"\n",
+ "\n",
+ "target_sampling_rate = 24000\n",
+ "raw_ds = raw_ds.cast_column(\"audio\", Audio(sampling_rate=target_sampling_rate))\n",
+ "\n",
+ "def preprocess_example(example):\n",
+ " conversation = [\n",
+ " {\n",
+ " \"role\": str(example[speaker_key]),\n",
+ " \"content\": [\n",
+ " {\"type\": \"text\", \"text\": example[\"text\"]},\n",
+ " {\"type\": \"audio\", \"path\": example[\"audio\"][\"array\"]},\n",
+ " ],\n",
+ " }\n",
+ " ]\n",
+ "\n",
+ " try:\n",
+ " model_inputs = processor.apply_chat_template(\n",
+ " conversation,\n",
+ " tokenize=True,\n",
+ " return_dict=True,\n",
+ " output_labels=True,\n",
+ " text_kwargs = {\n",
+ " \"padding\": \"max_length\", # pad to the max_length\n",
+ " \"max_length\": 256, # this should be the max length of audio\n",
+ " \"pad_to_multiple_of\": 8,\n",
+ " \"padding_side\": \"right\",\n",
+ " },\n",
+ " audio_kwargs = {\n",
+ " \"sampling_rate\": 24_000,\n",
+ " \"max_length\": 240001, # max input_values length of the whole dataset\n",
+ " \"padding\": \"max_length\",\n",
+ " },\n",
+ " common_kwargs = {\"return_tensors\": \"pt\"},\n",
+ " )\n",
+ " except Exception as e:\n",
+ " print(f\"Error processing example with text '{example['text'][:50]}...': {e}\")\n",
+ " return None\n",
+ "\n",
+ " required_keys = [\"input_ids\", \"attention_mask\", \"labels\", \"input_values\", \"input_values_cutoffs\"]\n",
+ " processed_example = {}\n",
+ " # print(model_inputs.keys())\n",
+ " for key in required_keys:\n",
+ " if key not in model_inputs:\n",
+ " print(f\"Warning: Required key '{key}' not found in processor output for example.\")\n",
+ " return None\n",
+ "\n",
+ " value = model_inputs[key][0]\n",
+ " processed_example[key] = value\n",
+ "\n",
+ "\n",
+ " # Final check (optional but good)\n",
+ " if not all(isinstance(processed_example[key], torch.Tensor) for key in processed_example):\n",
+ " print(f\"Error: Not all required keys are tensors in final processed example. Keys: {list(processed_example.keys())}\")\n",
+ " return None\n",
+ "\n",
+ " return processed_example\n",
+ "\n",
+ "processed_ds = raw_ds.map(\n",
+ " preprocess_example,\n",
+ " remove_columns=raw_ds.column_names,\n",
+ " desc=\"Preprocessing dataset\",\n",
+ ")"
]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "text = \"Sesame is a super cool TTS model which can be fine tuned with Unsloth.\"\n",
- "\n",
- "speaker_id = 0\n",
- "# Another equivalent way to prepare the inputs\n",
- "conversation = [\n",
- " {\"role\": str(speaker_id), \"content\": [{\"type\": \"text\", \"text\": text}]},\n",
- "]\n",
- "audio_values = model.generate(\n",
- " **processor.apply_chat_template(\n",
- " conversation,\n",
- " tokenize=True,\n",
- " return_dict=True,\n",
- " ).to(\"cuda\"),\n",
- " max_new_tokens=125, # 125 tokens is 10 seconds of audio, for longer speech increase this\n",
- " # play with these parameters to tweak results\n",
- " # depth_decoder_top_k=0,\n",
- " # depth_decoder_top_p=0.9,\n",
- " # depth_decoder_do_sample=True,\n",
- " # depth_decoder_temperature=0.9,\n",
- " # top_k=0,\n",
- " # top_p=1.0,\n",
- " # temperature=0.9,\n",
- " # do_sample=True,\n",
- " #########################################################\n",
- " output_audio=True\n",
- ")\n",
- "audio = audio_values[0].to(torch.float32).cpu().numpy()\n",
- "sf.write(\"example_without_context.wav\", audio, 24000)\n",
- "display(Audio(audio, rate=24000))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### Voice and style consistency\n",
- "\n",
- "Sesame CSM's power comes from providing audio context for each speaker. Let's pass a sample utterance from our dataset to ground speaker identity and style."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "zTIxxQqai7Jj"
+ },
+ "source": [
+ "\n",
+ "### Train the model\n",
+ "Now let's use Huggingface `Trainer`! More docs here: [Transformers docs](https://huggingface.co/docs/transformers/main_classes/trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "id": "G03LvbfOi7Jm"
+ },
+ "outputs": [],
+ "source": [
+ "from transformers import TrainingArguments, Trainer\n",
+ "from unsloth import is_bfloat16_supported\n",
+ "\n",
+ "trainer = Trainer(\n",
+ " model = model,\n",
+ " train_dataset = processed_ds,\n",
+ " args = TrainingArguments(\n",
+ " per_device_train_batch_size = 2,\n",
+ " gradient_accumulation_steps = 4,\n",
+ " warmup_steps = 5,\n",
+ " max_steps = 60,\n",
+ " learning_rate = 2e-4,\n",
+ " fp16 = not is_bfloat16_supported(),\n",
+ " bf16 = is_bfloat16_supported(),\n",
+ " logging_steps = 1,\n",
+ " optim = \"adamw_8bit\",\n",
+ " weight_decay = 0.01, # Turn this on if overfitting\n",
+ " lr_scheduler_type = \"linear\",\n",
+ " seed = 3407,\n",
+ " output_dir = \"outputs\",\n",
+ " report_to = \"none\", # Use this for WandB etc\n",
+ " ),\n",
+ ")"
+ ]
+ },
{
- "data": {
- "text/html": [
- "\n",
- " \n",
- " "
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "cellView": "form",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "-pEAe-cLi7Jo",
+ "outputId": "7493f8e2-a622-4c3a-82a1-fb64e517ff1f"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "GPU = Tesla T4. Max memory = 14.741 GB.\n",
+ "6.719 GB of memory reserved.\n"
+ ]
+ }
],
- "text/plain": [
- ""
+ "source": [
+ "# @title Show current memory stats\n",
+ "gpu_stats = torch.cuda.get_device_properties(0)\n",
+ "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
+ "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
+ "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
+ "print(f\"{start_gpu_memory} GB of memory reserved.\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000
+ },
+ "id": "szZOUYHgi7Js",
+ "outputId": "8727f1ec-cf16-4e88-84ef-fb0ff3521ae0"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1\n",
+ " \\\\ /| Num examples = 1,195 | Num Epochs = 1 | Total steps = 60\n",
+ "O^O/ \\_/ \\ Batch size per device = 2 | Gradient accumulation steps = 4\n",
+ "\\ / Data Parallel GPUs = 1 | Total batch size (2 x 4 x 1) = 8\n",
+ " \"-____-\" Trainable parameters = 29,032,448 of 1,661,132,609 (1.75% trained)\n"
+ ]
+ },
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Unsloth: Will smartly offload gradients to save VRAM!\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ ""
+ ],
+ "text/html": [
+ "\n",
+ " \n",
+ " \n",
+ "
\n",
+ " [60/60 05:39, Epoch 0/1]\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | Step | \n",
+ " Training Loss | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 1 | \n",
+ " 5.358000 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " 5.156600 | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " 5.326700 | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " 5.500300 | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " 5.252100 | \n",
+ "
\n",
+ " \n",
+ " | 6 | \n",
+ " 5.433400 | \n",
+ "
\n",
+ " \n",
+ " | 7 | \n",
+ " 5.189000 | \n",
+ "
\n",
+ " \n",
+ " | 8 | \n",
+ " 5.258200 | \n",
+ "
\n",
+ " \n",
+ " | 9 | \n",
+ " 5.611300 | \n",
+ "
\n",
+ " \n",
+ " | 10 | \n",
+ " 4.930900 | \n",
+ "
\n",
+ " \n",
+ " | 11 | \n",
+ " 5.350700 | \n",
+ "
\n",
+ " \n",
+ " | 12 | \n",
+ " 4.936600 | \n",
+ "
\n",
+ " \n",
+ " | 13 | \n",
+ " 4.824700 | \n",
+ "
\n",
+ " \n",
+ " | 14 | \n",
+ " 5.072300 | \n",
+ "
\n",
+ " \n",
+ " | 15 | \n",
+ " 4.765900 | \n",
+ "
\n",
+ " \n",
+ " | 16 | \n",
+ " 5.297400 | \n",
+ "
\n",
+ " \n",
+ " | 17 | \n",
+ " 5.151100 | \n",
+ "
\n",
+ " \n",
+ " | 18 | \n",
+ " 4.145100 | \n",
+ "
\n",
+ " \n",
+ " | 19 | \n",
+ " 5.055100 | \n",
+ "
\n",
+ " \n",
+ " | 20 | \n",
+ " 5.232600 | \n",
+ "
\n",
+ " \n",
+ " | 21 | \n",
+ " 4.815200 | \n",
+ "
\n",
+ " \n",
+ " | 22 | \n",
+ " 4.979900 | \n",
+ "
\n",
+ " \n",
+ " | 23 | \n",
+ " 5.096300 | \n",
+ "
\n",
+ " \n",
+ " | 24 | \n",
+ " 4.847400 | \n",
+ "
\n",
+ " \n",
+ " | 25 | \n",
+ " 5.031100 | \n",
+ "
\n",
+ " \n",
+ " | 26 | \n",
+ " 5.037100 | \n",
+ "
\n",
+ " \n",
+ " | 27 | \n",
+ " 4.599800 | \n",
+ "
\n",
+ " \n",
+ " | 28 | \n",
+ " 4.748800 | \n",
+ "
\n",
+ " \n",
+ " | 29 | \n",
+ " 4.657900 | \n",
+ "
\n",
+ " \n",
+ " | 30 | \n",
+ " 4.548700 | \n",
+ "
\n",
+ " \n",
+ " | 31 | \n",
+ " 5.342100 | \n",
+ "
\n",
+ " \n",
+ " | 32 | \n",
+ " 4.766600 | \n",
+ "
\n",
+ " \n",
+ " | 33 | \n",
+ " 5.078500 | \n",
+ "
\n",
+ " \n",
+ " | 34 | \n",
+ " 4.944400 | \n",
+ "
\n",
+ " \n",
+ " | 35 | \n",
+ " 4.821500 | \n",
+ "
\n",
+ " \n",
+ " | 36 | \n",
+ " 5.071400 | \n",
+ "
\n",
+ " \n",
+ " | 37 | \n",
+ " 5.042000 | \n",
+ "
\n",
+ " \n",
+ " | 38 | \n",
+ " 5.177700 | \n",
+ "
\n",
+ " \n",
+ " | 39 | \n",
+ " 4.640300 | \n",
+ "
\n",
+ " \n",
+ " | 40 | \n",
+ " 4.978900 | \n",
+ "
\n",
+ " \n",
+ " | 41 | \n",
+ " 4.730400 | \n",
+ "
\n",
+ " \n",
+ " | 42 | \n",
+ " 4.167100 | \n",
+ "
\n",
+ " \n",
+ " | 43 | \n",
+ " 4.744700 | \n",
+ "
\n",
+ " \n",
+ " | 44 | \n",
+ " 4.661600 | \n",
+ "
\n",
+ " \n",
+ " | 45 | \n",
+ " 4.835900 | \n",
+ "
\n",
+ " \n",
+ " | 46 | \n",
+ " 4.901400 | \n",
+ "
\n",
+ " \n",
+ " | 47 | \n",
+ " 5.343900 | \n",
+ "
\n",
+ " \n",
+ " | 48 | \n",
+ " 4.577700 | \n",
+ "
\n",
+ " \n",
+ " | 49 | \n",
+ " 5.101600 | \n",
+ "
\n",
+ " \n",
+ " | 50 | \n",
+ " 4.375900 | \n",
+ "
\n",
+ " \n",
+ " | 51 | \n",
+ " 4.889200 | \n",
+ "
\n",
+ " \n",
+ " | 52 | \n",
+ " 5.170700 | \n",
+ "
\n",
+ " \n",
+ " | 53 | \n",
+ " 4.451800 | \n",
+ "
\n",
+ " \n",
+ " | 54 | \n",
+ " 4.587500 | \n",
+ "
\n",
+ " \n",
+ " | 55 | \n",
+ " 5.225400 | \n",
+ "
\n",
+ " \n",
+ " | 56 | \n",
+ " 5.091400 | \n",
+ "
\n",
+ " \n",
+ " | 57 | \n",
+ " 4.855200 | \n",
+ "
\n",
+ " \n",
+ " | 58 | \n",
+ " 4.775500 | \n",
+ "
\n",
+ " \n",
+ " | 59 | \n",
+ " 5.003600 | \n",
+ "
\n",
+ " \n",
+ " | 60 | \n",
+ " 4.697800 | \n",
+ "
\n",
+ " \n",
+ "
"
+ ]
+ },
+ "metadata": {}
+ }
+ ],
+ "source": [
+ "trainer_stats = trainer.train()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "cellView": "form",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 218
+ },
+ "id": "SjWdtI8zi7Jv",
+ "outputId": "788b3c3f-5aec-4217-b7a7-7502fa0d5281"
+ },
+ "outputs": [
+ {
+ "output_type": "error",
+ "ename": "NameError",
+ "evalue": "name 'trainer_stats' is not defined",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipython-input-828055189.py\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mused_percentage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mround\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mused_memory\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mmax_memory\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mlora_percentage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mround\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mused_memory_for_lora\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mmax_memory\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{trainer_stats.metrics['train_runtime']} seconds used for training.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m print(\n\u001b[1;32m 8\u001b[0m \u001b[0;34mf\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mNameError\u001b[0m: name 'trainer_stats' is not defined"
+ ]
+ }
+ ],
+ "source": [
+ "# @title Show final memory and time stats\n",
+ "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
+ "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
+ "used_percentage = round(used_memory / max_memory * 100, 3)\n",
+ "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
+ "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
+ "print(\n",
+ " f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
+ ")\n",
+ "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
+ "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
+ "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
+ "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "JWCCkIXyi7Jy"
+ },
+ "source": [
+ "\n",
+ "### Inference\n",
+ "Let's run the model! You can change the prompts"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "5XGhdNfiihlx",
+ "outputId": "ab146fec-f0d9-4313-ce3d-cf51c848d8a7"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " \n",
+ " "
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from IPython.display import Audio, display\n",
+ "import soundfile as sf\n",
+ "\n",
+ "text = \"We just finished fine tuning a text to speech model... and it's pretty good!\"\n",
+ "speaker_id = 0\n",
+ "inputs = processor(f\"[{speaker_id}]{text}\", add_special_tokens=True).to(\"cuda\")\n",
+ "audio_values = model.generate(\n",
+ " **inputs,\n",
+ " max_new_tokens=125, # 125 tokens is 10 seconds of audio, for longer speech increase this\n",
+ " # play with these parameters to tweak results\n",
+ " # depth_decoder_top_k=0,\n",
+ " # depth_decoder_top_p=0.9,\n",
+ " # depth_decoder_do_sample=True,\n",
+ " # depth_decoder_temperature=0.9,\n",
+ " # top_k=0,\n",
+ " # top_p=1.0,\n",
+ " # temperature=0.9,\n",
+ " # do_sample=True,\n",
+ " #########################################################\n",
+ " output_audio=True\n",
+ ")\n",
+ "audio = audio_values[0].to(torch.float32).cpu().numpy()\n",
+ "sf.write(\"example_without_context.wav\", audio, 24000)\n",
+ "display(Audio(audio, rate=24000))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "xc6RJD6Hihlx",
+ "outputId": "f8fe4a14-e618-43d9-899d-2b0789c504b4"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " \n",
+ " "
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "text = \"Sesame is a super cool TTS model which can be fine tuned with Unsloth.\"\n",
+ "\n",
+ "speaker_id = 0\n",
+ "# Another equivalent way to prepare the inputs\n",
+ "conversation = [\n",
+ " {\"role\": str(speaker_id), \"content\": [{\"type\": \"text\", \"text\": text}]},\n",
+ "]\n",
+ "audio_values = model.generate(\n",
+ " **processor.apply_chat_template(\n",
+ " conversation,\n",
+ " tokenize=True,\n",
+ " return_dict=True,\n",
+ " ).to(\"cuda\"),\n",
+ " max_new_tokens=125, # 125 tokens is 10 seconds of audio, for longer speech increase this\n",
+ " # play with these parameters to tweak results\n",
+ " # depth_decoder_top_k=0,\n",
+ " # depth_decoder_top_p=0.9,\n",
+ " # depth_decoder_do_sample=True,\n",
+ " # depth_decoder_temperature=0.9,\n",
+ " # top_k=0,\n",
+ " # top_p=1.0,\n",
+ " # temperature=0.9,\n",
+ " # do_sample=True,\n",
+ " #########################################################\n",
+ " output_audio=True\n",
+ ")\n",
+ "audio = audio_values[0].to(torch.float32).cpu().numpy()\n",
+ "sf.write(\"example_without_context.wav\", audio, 24000)\n",
+ "display(Audio(audio, rate=24000))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "R7eEO7Wqihlx"
+ },
+ "source": [
+ "#### Voice and style consistency\n",
+ "\n",
+ "Sesame CSM's power comes from providing audio context for each speaker. Let's pass a sample utterance from our dataset to ground speaker identity and style."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "1omLYWKTihlx",
+ "outputId": "a5d86997-473f-491d-9a2c-08f341710cd9"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " \n",
+ " "
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "speaker_id = 0\n",
+ "\n",
+ "utterance = raw_ds[3][\"audio\"][\"array\"]\n",
+ "utterance_text = raw_ds[3][\"text\"]\n",
+ "text = \"Sesame is a super cool TTS model which can be fine tuned with Unsloth.\"\n",
+ "\n",
+ "# CSM will fill in the audio for the last text.\n",
+ "# You can even provide a conversation history back in as you generate new audio\n",
+ "\n",
+ "conversation = [\n",
+ " {\"role\": str(speaker_id), \"content\": [{\"type\": \"text\", \"text\": utterance_text},{\"type\": \"audio\", \"path\": utterance}]},\n",
+ " {\"role\": str(speaker_id), \"content\": [{\"type\": \"text\", \"text\": text}]},\n",
+ "]\n",
+ "\n",
+ "inputs = processor.apply_chat_template(\n",
+ " conversation,\n",
+ " tokenize=True,\n",
+ " return_dict=True,\n",
+ " )\n",
+ "audio_values = model.generate(\n",
+ " **inputs.to(\"cuda\"),\n",
+ " max_new_tokens=125, # 125 tokens is 10 seconds of audio, for longer text increase this\n",
+ " # play with these parameters to tweak results\n",
+ " # depth_decoder_top_k=0,\n",
+ " # depth_decoder_top_p=0.9,\n",
+ " # depth_decoder_do_sample=True,\n",
+ " # depth_decoder_temperature=0.9,\n",
+ " # top_k=0,\n",
+ " # top_p=1.0,\n",
+ " # temperature=0.9,\n",
+ " # do_sample=True,\n",
+ " #########################################################\n",
+ " output_audio=True\n",
+ ")\n",
+ "audio = audio_values[0].to(torch.float32).cpu().numpy()\n",
+ "sf.write(\"example_with_context.wav\", audio, 24000)\n",
+ "display(Audio(audio, rate=24000))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "OJIc7oYdi7J2"
+ },
+ "source": [
+ "\n",
+ "### Saving, loading finetuned models\n",
+ "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
+ "\n",
+ "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "RfyvxgUEi7J3",
+ "outputId": "6296c5b1-f7fc-47ba-dca5-77f74d89914c"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[]"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "model.save_pretrained(\"lora_model\") # Local saving\n",
+ "processor.save_pretrained(\"lora_model\")\n",
+ "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n",
+ "# processor.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving"
]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "speaker_id = 0\n",
- "\n",
- "utterance = raw_ds[3][\"audio\"][\"array\"]\n",
- "utterance_text = raw_ds[3][\"text\"]\n",
- "text = \"Sesame is a super cool TTS model which can be fine tuned with Unsloth.\"\n",
- "\n",
- "# CSM will fill in the audio for the last text.\n",
- "# You can even provide a conversation history back in as you generate new audio\n",
- "\n",
- "conversation = [\n",
- " {\"role\": str(speaker_id), \"content\": [{\"type\": \"text\", \"text\": utterance_text},{\"type\": \"audio\", \"path\": utterance}]},\n",
- " {\"role\": str(speaker_id), \"content\": [{\"type\": \"text\", \"text\": text}]},\n",
- "]\n",
- "\n",
- "inputs = processor.apply_chat_template(\n",
- " conversation,\n",
- " tokenize=True,\n",
- " return_dict=True,\n",
- " )\n",
- "audio_values = model.generate(\n",
- " **inputs.to(\"cuda\"),\n",
- " max_new_tokens=125, # 125 tokens is 10 seconds of audio, for longer text increase this\n",
- " # play with these parameters to tweak results\n",
- " # depth_decoder_top_k=0,\n",
- " # depth_decoder_top_p=0.9,\n",
- " # depth_decoder_do_sample=True,\n",
- " # depth_decoder_temperature=0.9,\n",
- " # top_k=0,\n",
- " # top_p=1.0,\n",
- " # temperature=0.9,\n",
- " # do_sample=True,\n",
- " #########################################################\n",
- " output_audio=True\n",
- ")\n",
- "audio = audio_values[0].to(torch.float32).cpu().numpy()\n",
- "sf.write(\"example_with_context.wav\", audio, 24000)\n",
- "display(Audio(audio, rate=24000))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "OJIc7oYdi7J2"
- },
- "source": [
- "\n",
- "### Saving, loading finetuned models\n",
- "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
- "\n",
- "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
},
- "id": "RfyvxgUEi7J3",
- "outputId": "6296c5b1-f7fc-47ba-dca5-77f74d89914c"
- },
- "outputs": [
{
- "data": {
- "text/plain": [
- "[]"
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "eve_aZkli7J9"
+ },
+ "source": [
+ "### Saving to float16\n",
+ "\n",
+ "We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {
+ "id": "-bAEpXHHi7J-",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 103,
+ "referenced_widgets": [
+ "9f2d0cf88629412ca46fa721940a1022",
+ "bf1226133372438ea7c313d8d3a05441",
+ "bac4095807f04c0480fc867e6b446197",
+ "f9a2c92879cf4f149dda57a9bcb5760f",
+ "0efa1da808894a25bf142f964b00dea7",
+ "d4cffa528d1f40149ba9f32aa1a2d83b",
+ "1c10862b0d284653a8a8d9707aa0b97b",
+ "25aa38fb97614893bda8270661791dd2",
+ "e4766f13e1d4487cbfaea305ddb5d9da",
+ "ef3619bf216b4e8a99460971c72eddaf",
+ "06b8433ac2414f1ca36ac60db983e65c"
+ ]
+ },
+ "outputId": "aa36e0a9-4c8e-4641-b602-eaa5dfb5c00f"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "No files have been modified since last commit. Skipping to prevent empty commit.\n",
+ "WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.\n"
+ ]
+ },
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Saved model to https://huggingface.co/NeuralNovel/csm\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "tokenizer.json: 0%| | 0.00/17.2M [00:00\n",
+ " \n",
+ " \n",
+ " \n",
+ "\n",
+ " Join Discord if you need help + ⭐️ Star us on Github ⭐️\n",
+ "\n"
]
- },
- "execution_count": 18,
- "metadata": {},
- "output_type": "execute_result"
}
- ],
- "source": [
- "model.save_pretrained(\"lora_model\") # Local saving\n",
- "processor.save_pretrained(\"lora_model\")\n",
- "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n",
- "# processor.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "eve_aZkli7J9"
- },
- "source": [
- "### Saving to float16\n",
- "\n",
- "We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {
- "id": "-bAEpXHHi7J-"
- },
- "outputs": [],
- "source": [
- "# Merge to 16bit\n",
- "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_16bit\",)\n",
- "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_16bit\", token = \"\")\n",
- "\n",
- "# Merge to 4bit\n",
- "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_4bit\",)\n",
- "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_4bit\", token = \"\")\n",
- "\n",
- "# Just LoRA adapters\n",
- "if False:\n",
- " model.save_pretrained(\"model\")\n",
- " tokenizer.save_pretrained(\"model\")\n",
- "if False:\n",
- " model.push_to_hub(\"hf/model\", token = \"\")\n",
- " tokenizer.push_to_hub(\"hf/model\", token = \"\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
- "\n",
- "Some other links:\n",
- "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n",
- "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n",
- "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n",
- "6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n",
- "\n",
- "\n",
- " \n",
- " \n",
- " \n",
- "\n",
- " Join Discord if you need help + \u2b50\ufe0f Star us on Github \u2b50\ufe0f\n",
- " |