> This is part 1 of 2 of the full documentation (pages 1–100 of 198).
> The content is paginated: fetch every part to see all of it.
> Next part: https://unsloth.ai/docs/llms-full.txt/1
> Page index: https://unsloth.ai/docs/llms.txt

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
This site has an agentic ask interface you may use to query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/index.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

---

# Unsloth Docs

Unsloth is an open-source framework for running and training LLMs.

Unsloth lets you run and train AI models on your own local hardware via an open-source UI.

Our docs will guide you through running & training your own LLM locally.

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download</a><a href="/docs/desktop#features" class="button secondary" data-icon="sparkles">Features</a><a href="https://github.com/unslothai/unsloth" class="button secondary" data-icon="github">GitHub</a>

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-cover data-type="image">Cover image</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>DeepSeek-V4.1-Flash</strong></td><td>Run the new October 17 model update!</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F4qwswsAZxmUaTEksSNJJ%2Fdeepseekv4.1.png?alt=media&amp;token=203263d6-e100-4c3f-9dd4-898391874d74">deepseekv4.1.png</a></td><td><a href="/docs/models/deepseek-v4.1-flash">DeepSeek-V4.1-Flash</a></td></tr><tr><td><strong>Qwen3.8-27B</strong></td><td>Run and train Qwen3.8-27B in Unsloth.</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FSQJGLfV6lTdcvKUuBsjC%2Fqwen3.8%20logo.png?alt=media&amp;token=7942746b-409d-4064-8e66-221fceb920dd">qwen3.8 logo.png</a></td><td><a href="/docs/models/qwen3.8">Qwen3.8-27B</a></td></tr><tr><td><strong>Qwen3.8-Flash-Next</strong></td><td>Run Meta's new SOTA open model locally.</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F02STQ7sMPvXPUVmCanee%2Fqwen%20next%20logo.png?alt=media&amp;token=b451d135-18c0-47e0-8584-ef64c30bc11f">qwen next logo.png</a></td><td><a href="/docs/models/qwen3.8-next">Qwen3.8-Flash-Next</a></td></tr><tr><td><strong>Decision model</strong></td><td>Turn any model into a decision model.</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FAR7yKT6SAX6lxfycVoHb%2Ftrain%20decision%20model%20thumbnail.png?alt=media&amp;token=0f302911-51dc-4d01-bf77-87433ad3490c">train decision model thumbnail.png</a></td><td><a href="/docs/basics/train-your-own-decision-model-with-unsloth">Train your own Decision Model</a></td></tr><tr><td><strong>NVIDIA Nemotron 3.5</strong></td><td>New 4B and 74B Bolt and Super VL models.</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F0WXsJqWMrGcNRAP5h4q7%2Fnvidia%20nemotron%203.5%20bolt.png?alt=media&amp;token=15a3c95c-9a2b-4f02-b7ad-ac11070b7a28">nvidia nemotron 3.5 bolt.png</a></td><td><a href="/docs/models/nemotron-3.5-bolt">NVIDIA Nemotron 3.5 Bolt</a></td></tr><tr><td><strong>Qwen-Image-2.1</strong></td><td>Run the new Turbo image model locally!</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FjBLXH1oRP52poQcN1pOn%2Fqwen%20image%202.1%20logo.png?alt=media&amp;token=ab285b72-146a-4409-b9ee-327fb78cb127">qwen image 2.1 logo.png</a></td><td><a href="/docs/models/qwen-image-2.1">Qwen-Image-2.1</a></td></tr></tbody></table>

### ⚡ Quickstart

Unsloth supports MacOS, Linux, [Windows](/docs/get-started/install/windows-installation), [NVIDIA](/docs/get-started/install/pip-install), [AMD](/docs/get-started/install/amd), Intel and CPU setups. See: [Unsloth Requirements](/docs/get-started/fine-tuning-for-beginners/unsloth-requirements). Download the native desktop app for your operating system:

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth Desktop</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, install Unsloth Studio natively, our browser-based web UI:

{% tabs %}
{% tab title="MacOS, Linux, WSL" %}
**Install Unsloth:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use the same command to update.
{% endtab %}

{% tab title="Windows" %}
**Install Unsloth:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

Use the same command to update.
{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}

{% columns %}
{% column width="50%" %}
{% content-ref url="/pages/BAeSP6aOxvSeDUzCgKOK" %}
[Complete LLM Directory](/docs/models/tutorials)
{% endcontent-ref %}

{% content-ref url="/pages/nw2c1elNySGBBav8WP9B" %}
[Fine-tuning Guide](/docs/get-started/fine-tuning-llms-guide)
{% endcontent-ref %}
{% endcolumn %}

{% column width="50%" %}
{% content-ref url="/pages/nQlzs5BcvqlaEjhsgbtY" %}
[Models](/docs/get-started/unsloth-model-catalog)
{% endcontent-ref %}

{% content-ref url="/pages/7sCtc6YWnJBYthTjQsr7" %}
[Unsloth API](/docs/basics/api)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

### 👾 Unsloth Start

{% columns %}
{% column width="58.333333333333336%" %}
[Unsloth Start](/docs/integrations/unsloth-start) lets you connect [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex) and other agents to local models via the `unsloth start` command.

Start Unsloth, load a model, open your project folder, and then run:

```bash
unsloth start claude
```

Replace `claude` with any agent below:
{% endcolumn %}

{% column width="41.666666666666664%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpXE6kCHjh8qOEaggf94M%2FScreenshot_20260718_122426.png?alt=media&amp;token=a59e4c8c-efdb-451b-b1f8-621955564f6d" alt="" width="563"><figcaption><p>Claude Code running with Qwen3.5 locally.</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

| Agent                                                              | Command                  |
| ------------------------------------------------------------------ | ------------------------ |
| <i class="fa-claude">:claude:</i> Claude Code                      | `unsloth start claude`   |
| <i class="fa-openai">:openai:</i> OpenAI Codex                     | `unsloth start codex`    |
| <i class="fa-vscode">:vscode:</i> Visual Studio Code               | `unsloth start vscode`   |
| <i class="fa-deepseek">:deepseek:</i> DeepSeek Harness             | `unsloth start dsh`      |
| <i class="fa-rectangle-vertical">:rectangle-vertical:</i> OpenCode | `unsloth start opencode` |
| <i class="fa-caduceus">:caduceus:</i> Hermes Agent                 | `unsloth start hermes`   |
| <i class="fa-lobster">:lobster:</i> OpenClaw                       | `unsloth start openclaw` |

### 🦥 Why Unsloth?

* We directly collab with teams behind [gpt-oss](https://docs.unsloth.ai/new/gpt-oss-how-to-run-and-fine-tune#unsloth-fixes-for-gpt-oss), [Qwen3](https://www.reddit.com/r/LocalLLaMA/comments/1kaodxu/qwen3_unsloth_dynamic_ggufs_128k_context_bug_fixes/), [Llama 4](https://github.com/ggml-org/llama.cpp/pull/12889), [Mistral](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B/discussions/18), [Gemma 1-3](https://news.ycombinator.com/item?id=39671146) and [Phi-4](https://unsloth.ai/blog/phi4), where we’ve **fixed critical bugs** that greatly improved model accuracy. Andrej Karpathy for example has [praised our work](https://x.com/karpathy/status/1765473722985771335).
* Unsloth streamlines local training, inference, data, and deployment
* Unsloth supports inference and training for 500+ models: [vision](/docs/basics/vision-fine-tuning), [TTS](/docs/basics/text-to-speech-tts-fine-tuning), [embedding](/docs/basics/embedding-finetuning), [RL](/docs/get-started/reinforcement-learning-rl-guide)

### ⭐ Features

Unsloth lets you run and train models for text, [audio](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning), [embedding](https://unsloth.ai/docs/new/embedding-finetuning), [vision](https://unsloth.ai/docs/basics/vision-fine-tuning) and more. Unsloth provides many key features for both inference and training:

#### Inference

* Run and train LLMs, diffusion, embedding, audio models: [Qwen3.8](/docs/models/qwen3.8), [Kimi K3](https://unsloth.ai/docs/models/kimi-k3), MiniMax-H3, [Muse Glimmer](https://unsloth.ai/docs/models/muse-glimmer), [DeepSeek-V4](https://unsloth.ai/docs/models/deepseek-v4), [Gemma 4](https://unsloth.ai/docs/models/gemma-4).
* **Agents & Tools:** Use local models with [Claude Code](https://unsloth.ai/docs/basics/claude-code), [Codex](https://unsloth.ai/docs/basics/codex), and [MCP](https://unsloth.ai/docs/basics/mcp), including tool calling and code execution.
* **Search & RAG:** Use private and unlimited web search, deep research, auto-compaction (rolling context window) and RAG.
* **Image and video:** Run and train [image](https://unsloth.ai/docs/basics/diffusion-image) and video diffusion or multimodal models
* **Remote & LAN:** Access your local models from any device on [LAN](https://unsloth.ai/docs/basics/lan) or remotely through secure [Cloudflare](https://unsloth.ai/docs/basics/how-to-serve-local-llms-anywhere-secure-remote-access-with-cloudflare-and-unsloth) HTTPS.
* **Connect:** Serve models through an [OpenAI compatible API](https://unsloth.ai/docs/basics/api). Also connect your ChatGPT/Codex subscription and [cloud providers](https://unsloth.ai/docs/integrations/connections)

#### Train & Deploy

* **Fine-tuning:** Train LLMs, diffusion, TTS, and embedding models 2× faster with 70% less VRAM with [no accuracy loss](https://unsloth.ai/blog#training)
* **Complete support:** Supports [reinforcement learning](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide), LoRA, QLoRA, full fine tuning, pretraining, RL, GRPO, DPO, and FP8.
* **Export & Deploy:** [Export](https://unsloth.ai/docs/new/studio/export) or Deploy models with including [GGUF](https://unsloth.ai/docs/basics/inference-and-deployment/saving-to-gguf), NVFP4, FP8 and more formats.
* **Datasets:** Build datasets from PDFs, CSVs, DOCX files, and more with [Data Recipes](https://unsloth.ai/docs/new/studio/data-recipe).

### **Video Demo**

{% embed url="<https://www.youtube.com/watch?v=_1hgTTuber4>" %}

{% columns %}
{% column width="50%" %}
{% content-ref url="/pages/OktH76Rsg2WQ12B4KR5H" %}
[Unsloth Updates](/docs/new/changelog)
{% endcontent-ref %}

{% content-ref url="/pages/gEugERiAw2ztDNt98JVR" %}
[Inference & Deployment](/docs/basics/inference-and-deployment)
{% endcontent-ref %}
{% endcolumn %}

{% column width="50%" %}
{% content-ref url="/pages/55c8159250801d9ea2207b9eb88a9a11b331d3ad" %}
[Unsloth Start](/docs/integrations/unsloth-start)
{% endcontent-ref %}

{% content-ref url="/pages/QznsvWxKKvrY6PdiByzz" %}
[Dynamic 3.0 GGUFs](/docs/basics/dynamic-3.0-ggufs)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-134302f2507d4313b9575917c9a43b0a0028856c%2Flarge%20sloth%20wave.png?alt=media" alt="" width="188"><figcaption></figcaption></figure>


# Introducing Unsloth Desktop

Unsloth Desktop is a free, open-source app for Mac, Windows and Linux.

<h2 align="center"><strong>Introducing Unsloth Desktop</strong></h2>

<p align="center"><strong>Unsloth Desktop (Beta)</strong> is a free, <strong>open-source</strong> app for running and training AI models on your own local hardware. Available for macOS, Windows, and Linux.</p>

<p align="center">Unsloth lets you run, train and deploy LLMs, <strong>diffusion</strong> image/video, <strong>MLX</strong>, GGUF and audio models.</p>

<p align="center"><a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download</a><a href="/docs/desktop#feature-deep-dive" class="button secondary" data-icon="sparkles">Features</a><a href="https://github.com/unslothai/unsloth" class="button secondary" data-icon="github">GitHub</a></p>

{% columns %}
{% column %}

{% endcolumn %}

{% column width="83.33333333333333%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHTlw8oO7wDIYyGqilty0%2Funsloth%20qwen3.8%20final.png?alt=media&#x26;token=b8b32558-d0c8-4de8-bedf-7ad10e2c4c4a" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="8.333333333333314%" %}

{% endcolumn %}
{% endcolumns %}

<h2 align="center"><strong>Features</strong> ✨</h2>

<p align="center"><strong>Explore all features in our Feature Deep Dive:</strong> <a href="#feature-deep-dive" class="button primary">Feature Deep Dive -></a></p>

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="image">Cover image</th></tr></thead><tbody><tr><td><h3>Accurate tool calls</h3></td><td>Get 50% more accurate, self-healing tool calls and secure code execution. <strong>Parallel chatting</strong> is also supported.</td><td><a href="/docs/new/studio/chat">Chat with models</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLL8RSnAuc7XdutjAVD4n%2Fcode%20exec.png?alt=media&#x26;token=566a7385-69e0-46db-a022-bef8e2cad677">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLL8RSnAuc7XdutjAVD4n%2Fcode%20exec.png?alt=media&#x26;token=566a7385-69e0-46db-a022-bef8e2cad677</a></td></tr><tr><td><h3>Diffusion image/video</h3></td><td>Run and train image/video diffusion models like Qwen-Image-2.1<strong>,</strong> MiniMax H3 with faster inference.</td><td><a href="#image-and-video-generation">#image-and-video-generation</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHD8yIrQN6pS7qCNYMVxp%2Ffklrfklf.gif?alt=media&#x26;token=9e7daea2-c9cc-404c-af53-325833224c94">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHD8yIrQN6pS7qCNYMVxp%2Ffklrfklf.gif?alt=media&#x26;token=9e7daea2-c9cc-404c-af53-325833224c94</a></td></tr><tr><td><h3>Make your agent local</h3></td><td>Connect local LLMs to agentic tools like Claude Code and Codex with <strong>model swapping.</strong></td><td><a href="/docs/integrations/unsloth-start">Unsloth Start</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FaXmr0nHDKoSkhX6ESjLz%2Fclaude%20code%20image.png?alt=media&#x26;token=67e609e9-b329-4c19-9e4d-2aed0fe34f9c">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FaXmr0nHDKoSkhX6ESjLz%2Fclaude%20code%20image.png?alt=media&#x26;token=67e609e9-b329-4c19-9e4d-2aed0fe34f9c</a></td></tr><tr><td><h3>Unlimited Web search</h3></td><td>Get unlimited, private and secure web search and deep research</td><td><a href="/docs/new/studio#web-search-upgraded">Unsloth Studio</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6hy8X5V7FVW0NSWJfQAj%2Fwebsearch%20nice.png?alt=media&#x26;token=8a02b6ac-3c85-484f-9bc7-f7d3eee24cd9">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6hy8X5V7FVW0NSWJfQAj%2Fwebsearch%20nice.png?alt=media&#x26;token=8a02b6ac-3c85-484f-9bc7-f7d3eee24cd9</a></td></tr><tr><td><h3>Run the latest models</h3></td><td>Discover, manage and train the latest Qwen3.8-27B, MiniMax-H3, Muse Glimmer, Kimi K3 models.</td><td><a href="/docs/models/tutorials">Complete LLM Directory</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FTSjv63MOktgx6PbgHAi2%2Fmodel%20hub%20models.png?alt=media&#x26;token=b884e283-ff8b-4d4c-b4cb-7864d4335aa0">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FTSjv63MOktgx6PbgHAi2%2Fmodel%20hub%20models.png?alt=media&#x26;token=b884e283-ff8b-4d4c-b4cb-7864d4335aa0</a></td></tr><tr><td><h3><strong>Deploy anywhere</strong></h3></td><td>Serve models on your local network, or expose them anywhere over secure Cloudflare HTTPS or <a href="/docs/basics/lan">LAN</a>.</td><td><a href="/docs/basics/how-to-serve-local-llms-anywhere-secure-remote-access-with-cloudflare-and-unsloth">Serve Models with HTTPS</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJDIIEQ8fFZy2g5snC2kp%2Fcloudflare%20url.png?alt=media&#x26;token=aadb42b5-16f0-42d1-8059-c2611e7cd931">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJDIIEQ8fFZy2g5snC2kp%2Fcloudflare%20url.png?alt=media&#x26;token=aadb42b5-16f0-42d1-8059-c2611e7cd931</a></td></tr></tbody></table>

<h2 align="center"><strong>Get started</strong> 🦥</h2>

<p align="center">Unsloth is a <strong>Tauri based</strong> app. Install it, download models and start chatting!</p>

<p align="center"><strong>Download Unsloth Desktop:</strong><a href="https://unsloth.ai/download/mac" class="button primary" data-icon="apple">macOS</a><a href="https://unsloth.ai/download/windows" class="button primary" data-icon="windows">Windows</a><a href="https://unsloth.ai/download/linux" class="button primary" data-icon="linux">Linux and WSL</a></p>

{% columns %}
{% column width="16.666666666666664%" %}

{% endcolumn %}

{% column width="66.66666666666667%" %}
{% stepper %}
{% step %}

### Install Unsloth Desktop <a href="#install-jan" id="install-jan"></a>

1. Download [Unsloth Desktop](https://unsloth.ai/download)
2. Install the app for [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), [Linux](/docs/get-started/install/linux)
3. Launch the app
   {% endstep %}

{% step %}

### Choose a model

Open 'Select model' dropdown on top or 'Model hub' tab, choose a model and a quantization that fits your device, then download it. Once it finishes, start chatting - no setup required.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLMeRGP8AuuS5X4LA27kS%2FScreenshot%202026-08-08%20at%207.46.09%E2%80%AFAM.png?alt=media&#x26;token=16c3dfbb-3c98-47a2-9be3-ce77f3e9b806" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

### Unsloth is now ready

To chat, type a message and press Enter.

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FTSjv63MOktgx6PbgHAi2%2Fmodel%20hub%20models.png?alt=media&#x26;token=b884e283-ff8b-4d4c-b4cb-7864d4335aa0" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}
{% endcolumn %}

{% column width="16.666666666666657%" %}

{% endcolumn %}
{% endcolumns %}

<h2 align="center"><strong>Feature Deep Dive ⭐</strong></h2>

<p align="center">See everything Unsloth Desktop has to offer:</p>

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmIpCCk1g8Y0K45dltuRc%2FScreenshot%202026-08-12%20at%202.06.00%E2%80%AFAM.png?alt=media&#x26;token=b3408f39-9bab-41c3-834c-51692e1e4587" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="41.666666666666686%" %}

## <i class="fa-shield">:shield:</i> Control permissions

Unsloth, like Claude Code and Codex, provides permission controls for any model that uses tool calls. This means the model and Unsloth cannot access, modify, or edit your files or use the internet without your approval.

You can choose the level of access you want: Unsloth can run tools inside a secure environment to protect your system, or, if you prefer, you can allow it to directly access and edit your files.
{% endcolumn %}

{% column %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.666666666666664%" %}

## <i class="fa-globe-pointer">:globe-pointer:</i> **Any Endpoint. Every Tool.**

Unsloth removes the barrier between your inference setup and your workflow. Whether your model is running through Unsloth’s inference or your own remote OpenAI-compatible endpoint, you can give it access to Unsloth’s full suite of tools, including web search, code execution, deep research, and more.
{% endcolumn %}

{% column width="41.666666666666664%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FOTsFpguwJsfJMum5rWr0%2FScreenshot%202026-08-16%20at%202.35.13%E2%80%AFAM.png?alt=media&#x26;token=e566f3f9-3012-467d-8163-0fbbba3de7ba" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="8.333333333333332%" %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlzR5PrKmc1UT3jVlWOjm%2Fmobile_studio1_cropped(1).gif?alt=media&#x26;token=3df768c6-07d8-40c6-9c02-a881547d44bd" alt="" width="180"><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="41.666666666666686%" %}

## <i class="fa-comment">:comment:</i> Access model anywhere

Serve your local or over HTTPS through Unsloth's free [Cloudflare tunnel](/docs/basics/how-to-serve-local-llms-anywhere-secure-remote-access-with-cloudflare-and-unsloth). Check a run from **your phone**, your laptop, or anywhere else you happen to be.

Bind the app to your network with `-H 0.0.0.0`, or open a free Cloudflare tunnel for HTTPS.
{% endcolumn %}

{% column %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.66666666666666%" %}

## <i class="fa-terminal">:terminal:</i> Code execution

Get up to 50% more accurate tool-calling with self-healing tool calls that detect, repair and retry failures automatically.

Execute Bash and Python in a secure environment. Unsloth offers more reliable call termination to reduce loops, better healing and deduplication to prevent XML leakage.
{% endcolumn %}

{% column width="41.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F2vV1as2G5j736pz8F6Eq%2Fcode%20exec%20vid.gif?alt=media&#x26;token=17eaa810-9901-4ac0-9de5-4a3e62616ccd" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FcMQ75NHKaf3xosAL87cq%2Fremote-lan-settings2.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="41.66666666666667%" %}

## <i class="fa-wifi">:wifi:</i> Serve on your local network

Put Unsloth on your Wi-Fi or wired network and [serve the Unlosth over LAN](/docs/basics/lan). Traffic never leaves your network, so latency is low, and it works with no internet at all.

Press **Start** on the **LAN access** card, or launch with `-H 0.0.0.0`.
{% endcolumn %}

{% column width="8.333333333333336%" %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

## <i class="fa-image">:image:</i> **Image and Video Generation**

For MiniMax-H3 FP8 on an NVIDIA B200, a 960×544, 124-frame, 8-step generation dropped from 70+ seconds to **13 seconds**. Generate with [Qwen-Image-2.1](/docs/models/qwen-image-2.1)**,** FLUX, Z-Image, LTX, Wan and fine-tuned LoRA adapters.

Transform, inpaint, extend, upscale, reference and edit existing images.
{% endcolumn %}

{% column width="41.666666666666664%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYKI8L9axIJ7OVPOzCRv1%2Fimage%20diffusion.gif?alt=media&#x26;token=9925f476-2eba-4809-abc8-5b812c1e52ca" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="8.333333333333325%" %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

## <i class="fa-rectangle-api">:rectangle-api:</i> Serve models via API

You can use `unsloth run` command and serve models like [Qwen3.8](/docs/models/qwen3.8) and change any settings, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api).
{% endcolumn %}

{% column width="49.99999999999999%" %}
See an example of using Unsloth API with Qwen3.8:

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/Qwen3.8-27B-GGUF-GGUF:UD-Q4_K_XL
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --chat-template-kwargs '{"reasoning_effort":"medium"}'
```

{% endcode %}
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

## <i class="fa-image">:image:</i> Use the Latest models

You can run and train nearly every model, including upcoming ones. Expect Day Zero support for models like Qwen3.8, Gemma, Meta, NVIDIA, GLM, Gemma models and more. It's all thanks to llama.cpp and Hugging Face and we're also proud to contribute back to the ecosystem.
{% endcolumn %}

{% column width="49.99999999999999%" %}

<table data-card-size="large" data-column-title-hidden data-card-wrap="false" data-view="cards"><thead><tr><th></th><th data-hidden data-card-cover data-type="image">Cover image</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><h3>Qwen3.8</h3></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FSQJGLfV6lTdcvKUuBsjC%2Fqwen3.8%20logo.png?alt=media&#x26;token=7942746b-409d-4064-8e66-221fceb920dd">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FSQJGLfV6lTdcvKUuBsjC%2Fqwen3.8%20logo.png?alt=media&#x26;token=7942746b-409d-4064-8e66-221fceb920dd</a></td><td><a href="/docs/models/qwen3.8">Qwen3.8-27B</a></td></tr><tr><td><h4>Meta Muse Glimmer</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fnbdjif7yfvvH1j6aBmpS%2FMuse%20glimmer%20logo%20final.png?alt=media&#x26;token=5031f9dc-c257-4488-95be-856e1ea31ec8">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fnbdjif7yfvvH1j6aBmpS%2FMuse%20glimmer%20logo%20final.png?alt=media&#x26;token=5031f9dc-c257-4488-95be-856e1ea31ec8</a></td><td></td></tr><tr><td><h4>Kimi K3</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwCzVjy7Q6EQLthkUuZqk%2Fkimik3%20logo.png?alt=media&#x26;token=945dfb54-9c25-4539-9138-e23f841ca168">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwCzVjy7Q6EQLthkUuZqk%2Fkimik3%20logo.png?alt=media&#x26;token=945dfb54-9c25-4539-9138-e23f841ca168</a></td><td><a href="/docs/models/kimi-k3">Kimi K3 </a></td></tr><tr><td><h4>DeepSeek V4</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYCbdIeMsEQmDJoyii4tl%2Fdeepseek%20v4%20logo.png?alt=media&#x26;token=0d1ec333-bcbd-4ecb-8891-5393a1c5bb0a">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYCbdIeMsEQmDJoyii4tl%2Fdeepseek%20v4%20logo.png?alt=media&#x26;token=0d1ec333-bcbd-4ecb-8891-5393a1c5bb0a</a></td><td><a href="/docs/models/deepseek-v4">DeepSeek-V4-Flash</a></td></tr><tr><td><h4>Qwen3.6</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fdy7z03AUFzXHKqqOY7kC%2Fqwen3.6%20logo.png?alt=media&#x26;token=a894c047-a9ea-4f9c-824a-be86ec81f54d">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fdy7z03AUFzXHKqqOY7kC%2Fqwen3.6%20logo.png?alt=media&#x26;token=a894c047-a9ea-4f9c-824a-be86ec81f54d</a></td><td><a href="/docs/models/qwen3.6">Qwen3.6</a></td></tr><tr><td><h4>GLM-5.2</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmbYXj0v0p5zbeESPDYUr%2Fglm52.png?alt=media&#x26;token=0cb4ae58-d249-403a-9fb5-72e9e93f8da9">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmbYXj0v0p5zbeESPDYUr%2Fglm52.png?alt=media&#x26;token=0cb4ae58-d249-403a-9fb5-72e9e93f8da9</a></td><td><a href="/docs/models/glm-5.2">GLM-5.2</a></td></tr><tr><td><h4>Gemma 4</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkEjWOJqBWCtIN9Cg6CdI%2FGemma%204%20landscape.png?alt=media&#x26;token=57d3f596-dae8-4eab-80e6-0847794ffc8d">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkEjWOJqBWCtIN9Cg6CdI%2FGemma%204%20landscape.png?alt=media&#x26;token=57d3f596-dae8-4eab-80e6-0847794ffc8d</a></td><td><a href="/docs/models/gemma-4">Gemma 4</a></td></tr><tr><td><h4>MiniMax M3</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsaXyQq0XeFuudTK5ZEGw%2Fminimaxm3.png?alt=media&#x26;token=4af28954-abd2-4163-a0fe-ed4e42e25d2f">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsaXyQq0XeFuudTK5ZEGw%2Fminimaxm3.png?alt=media&#x26;token=4af28954-abd2-4163-a0fe-ed4e42e25d2f</a></td><td><a href="/docs/models/minimax-m3">MiniMax M3</a></td></tr><tr><td><h4>DiffusionGemma</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQNex1tTcNU8rPfsa1Pja%2Fdiffusiongemma%20logo.png?alt=media&#x26;token=37e9fa55-c003-4877-9519-d1687ce48322">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQNex1tTcNU8rPfsa1Pja%2Fdiffusiongemma%20logo.png?alt=media&#x26;token=37e9fa55-c003-4877-9519-d1687ce48322</a></td><td><a href="/docs/models/diffusiongemma">DiffusionGemma</a></td></tr></tbody></table>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.666666666666664%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZmacD0fexC9rfEGFjsY4%2Fdeepresearch-unsloth.gif?alt=media&#x26;token=c885223e-6776-4267-aac9-70c4a2f8b83c" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="41.66666666666667%" %}

## <i class="fa-magnifying-glass">:magnifying-glass:</i> Deep Research

Normal search utilises the web for its answers and happens while the model is still thinking.

Deep research plans first, searches for the best sources, then produces a detailed report with all citations.
{% endcolumn %}

{% column width="8.33333333333334%" %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="41.66666666666666%" %}

## <i class="fa-cloud">:cloud:</i> Connect Providers

You can connect Unsloth directly to your ChatGPT/Codex subscription.

Run models from OpenAI, Anthropic, Ollama, llama.cpp, vLLM, and more.

Use the same Unsloth chat interface for local and cloud models with support for tool-calling, image gen, [prompt caching](https://sites.gitbook.com/preview/site_mXXTe/~/revisions/KsWgqmEjr66G4kYiZdG8#prompt-caching) to reduce token usage while preserving provider-native features like OpenAI’s [web search](https://sites.gitbook.com/preview/site_mXXTe/~/revisions/KsWgqmEjr66G4kYiZdG8#web-search-and-thinking) and [code execution](https://sites.gitbook.com/preview/site_mXXTe/~/revisions/KsWgqmEjr66G4kYiZdG8#code-execution).
{% endcolumn %}

{% column width="41.66666666666667%" %}
{% embed url="<https://github.com/user-attachments/assets/70456395-e016-4273-8256-35adb206267e>" %}
{% endcolumn %}

{% column width="8.333333333333336%" %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FO8pfkKOJeIjxnBdKgtO8%2Fimage.png?alt=media&#x26;token=5932d6f2-ff7e-4ea5-8ac4-46da1b294a66" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="41.66666666666667%" %}

## <i class="fa-image">:image:</i> **Train Diffusion Models**

Train LoRA adapters for SDXL, FLUX.2, Qwen-Image and Z-Image on your own images. Caption them in Unsloth, pick a rank, and hit train. Then you can export it and load the model back in Unsloth for inference.
{% endcolumn %}

{% column %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

{% endcolumn %}

{% column width="41.66666666666666%" %}

## <i class="fa-flask">:flask:</i> Train with no code

Drop in a PDF, CSV or JSON and go. LoRA, full fine-tuning, pretraining. All 2x faster, 70% less VRAM, no accuracy loss. Multi-GPU and latest models work.

Text, diffusion, [audio](/docs/basics/text-to-speech-tts-fine-tuning) and [image](/docs/basics/vision-fine-tuning) models are supported!
{% endcolumn %}

{% column width="41.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FD7qpbGJLY5YMv0E0QgrB%2Ftrainingrun_LLM-unsloth.gif?alt=media&#x26;token=feb766a7-1a7b-4f47-b5f4-b38e3a0d1470" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="8.333333333333353%" %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FiMHF9Mmbt4BstcwHwfzY%2FScreenshot%202026-08-11%20at%205.24.10%E2%80%AFAM.png?alt=media&#x26;token=b8390a91-32bc-4831-bba3-4f05a3f47d7e" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="41.66666666666666%" %}

## <i class="fa-microphone">:microphone:</i> Transcribe/generate audio

Generate, fine-tune or transcribe audio with Unsloth completely locally. Text-to-speech, speech-to-text, Whisper, Qwen3-ASR, you name it!
{% endcolumn %}

{% column width="8.333333333333336%" %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="41.66666666666667%" valign="middle" %}

## <i class="fa-alien-8bit">:alien-8bit:</i> Connect your Agent

[Unsloth Start](/docs/integrations/unsloth-start) lets you connect [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex) and other agents to local models via the `unsloth start` command.

Start Unsloth, load a model, open your project folder, and then run:

```bash
unsloth start claude
```

{% endcolumn %}

{% column width="49.99999999999997%" %}

<table data-card-size="large" data-column-title-hidden data-card-wrap="false" data-view="cards"><thead><tr><th></th><th data-hidden data-card-cover data-type="image">Cover image</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><h4>Claude Code</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FT8nv3UJzIrcxmjSn8OfW%2Fclaude-code.webp?alt=media&#x26;token=811f9635-7321-427c-a11e-29a40e49e376">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FT8nv3UJzIrcxmjSn8OfW%2Fclaude-code.webp?alt=media&#x26;token=811f9635-7321-427c-a11e-29a40e49e376</a></td><td><a href="/docs/basics/claude-code">Claude Code</a></td></tr><tr><td><h4>Codex</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLxRuVHEBYywquhRGBYu0%2Fcodex%20only%20logo.png?alt=media&#x26;token=7faa9cbd-090a-45dc-9d5e-acb1a9591f27">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLxRuVHEBYywquhRGBYu0%2Fcodex%20only%20logo.png?alt=media&#x26;token=7faa9cbd-090a-45dc-9d5e-acb1a9591f27</a></td><td><a href="/docs/basics/codex">OpenAI Codex</a></td></tr><tr><td>VS Code</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsBmg9gZ062wIQDAuibJ7%2Fvscode-card.png?alt=media&amp;token=1659960f-0876-4469-a6a7-7408e7a6570a">vscode-card.png</a></td><td><a href="/docs/integrations/how-to-run-local-ai-models-in-vs-code">Run models in VS Code</a></td></tr><tr><td><h4>OpenCode</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlCi6Ab4f2qLlNj2EWGL0%2Fopencodelogo.png?alt=media&#x26;token=6118773e-5392-4718-a431-a50157a34cbe">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlCi6Ab4f2qLlNj2EWGL0%2Fopencodelogo.png?alt=media&#x26;token=6118773e-5392-4718-a431-a50157a34cbe</a></td><td><a href="/docs/integrations/opencode">OpenCode</a></td></tr><tr><td><h4>Hermes Agent</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6ysPTzsXIO5diu4vOTCh%2Fimages.jpeg?alt=media&#x26;token=5cc58141-69eb-4450-8214-7a8421fbe2a5">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6ysPTzsXIO5diu4vOTCh%2Fimages.jpeg?alt=media&#x26;token=5cc58141-69eb-4450-8214-7a8421fbe2a5</a></td><td><a href="/docs/integrations/hermes-agent">Hermes Agent</a></td></tr><tr><td><h4>Unsloth API</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoRer1Vbuh9wSPTawz0ey%2Funsloth%20api%20long%20logo.png?alt=media&#x26;token=860a8143-9d59-466b-abed-6ea7d67e51fd">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoRer1Vbuh9wSPTawz0ey%2Funsloth%20api%20long%20logo.png?alt=media&#x26;token=860a8143-9d59-466b-abed-6ea7d67e51fd</a></td><td><a href="/docs/basics/api">Unsloth API</a></td></tr><tr><td><h4>OpenClaw</h4></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FAd5uo6LjavtBrrKZfwZ3%2Fopenclaw-hero-light.png?alt=media&#x26;token=3097e40f-e80c-4c9c-8efc-02ecf5530ef6">https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FAd5uo6LjavtBrrKZfwZ3%2Fopenclaw-hero-light.png?alt=media&#x26;token=3097e40f-e80c-4c9c-8efc-02ecf5530ef6</a></td><td><a href="/docs/integrations/openclaw">OpenClaw</a></td></tr></tbody></table>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="83.33333333333334%" %}

<h2 align="center"><strong>Video Demo</strong></h2>

{% embed url="<https://www.youtube.com/watch?v=_1hgTTuber4>" %}
{% endcolumn %}

{% column %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="83.33333333333334%" %}

<h2 align="center">Frequently asked questions</h2>
{% endcolumn %}

{% column %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="41.66666666666667%" %}

#### Do you collect my data?

No telemetry. Unsloth detects your GPU type and device so the app can know what works. The app can run entirely offline.

#### Can I use models I already downloaded?

Yes, they are found automatically. If yours are not you can specify your own custom folders.

#### Why is inference slower sometimes?

Web search, code execution and tool-call healing all cost time. Turn them off and speed should match any other llama.cpp app. Still slow? Open a GitHub issue.
{% endcolumn %}

{% column width="41.66666666666667%" %}

#### GPU only?

No. Unsloth works on a wide variety of CPU, Mac etc setups.

#### Does it support OpenAI-compatible APIs?

Yes. See the [API guide](/docs/basics/api). We also support connection to [Cloud models](/docs/integrations/connections) or other APIs like Anthropic or OpenAI.

#### What devices does Unsloth support?

Unsloth supports all OS including Mac, Windows, Linux and WSL and supports NVIDIA, Intel, AMD and Mac GPUs/CPUs. Older hardware however may not be well supported.
{% endcolumn %}

{% column %}

{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="8.333333333333332%" %}

{% endcolumn %}

{% column width="83.33333333333333%" %}
A huge thank you to NVIDIA and Hugging Face for being part of our launch. Also thanks to all of our early beta testers for Unsloth Desktop, we truly appreciate your time and feedback. We’d also like to thank Jan for inspiration and llama.cpp, PyTorch, stablediffusion.cpp, and open model labs for providing the infrastructure that made Unsloth Desktop possible.
{% endcolumn %}

{% column width="8.333333333333321%" %}

{% endcolumn %}
{% endcolumns %}


# Unsloth Installation

Install Unsloth Desktop, Unsloth Studio, or Unsloth Core.

Unsloth can be installed in three distinct ways: [**Unsloth Desktop**](/docs/desktop), the native app; [**Unsloth Studio**](/docs/new/studio), the browser-based web UI; or **Unsloth Core**, the original code-based Python package.

### Unsloth Desktop

Download the native desktop app for your operating system:

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

### Unsloth Studio

**Unsloth Studio** is our web-based UI. Follow the instructions below to install it natively:

{% tabs %}
{% tab title="MacOS, Linux, WSL" %}
**Install Unsloth:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use the same command to update.
{% endtab %}

{% tab title="Windows" %}
**Install Unsloth:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

Use the same command to update.
{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
**Install Unsloth:**

```bash
brew install --cask unsloth
```

Update Unsloth:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
**Install Unsloth:**

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth:

<pre class="language-powershell"><code class="lang-powershell"><strong>winget upgrade --id UnslothAI.Unsloth -e
</strong></code></pre>

{% endtab %}
{% endtabs %}

For requirements and advanced options, see the [Unsloth Studio installation guide](https://unsloth.ai/docs/new/studio/install).

### Unsloth Core

**Unsloth Core** is the original Python package for code-based training and inference. Follow the [uv, pip, and virtual environment guide](https://unsloth.ai/docs/get-started/install/pip-install).

<table data-view="cards"><thead><tr><th data-type="content-ref"></th><th data-hidden data-card-cover data-type="image">Cover image</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><a href="/docs/get-started/install/mac">MacOS</a></td><td></td><td><a href="/docs/get-started/install/mac">MacOS</a></td></tr><tr><td><a href="/docs/get-started/install/windows-installation">Windows</a></td><td></td><td><a href="/docs/get-started/install/windows-installation">Windows</a></td></tr><tr><td><a href="/docs/get-started/install/pip-install">uv, pip install &amp; venv</a></td><td></td><td><a href="/docs/get-started/install/pip-install">uv, pip install &amp; venv</a></td></tr><tr><td><a href="/docs/get-started/install/docker">Docker</a></td><td></td><td><a href="/docs/get-started/install/docker">Docker</a></td></tr><tr><td><a href="/docs/get-started/install/linux">Linux</a></td><td></td><td><a href="/docs/get-started/install/linux">Linux</a></td></tr><tr><td><a href="/docs/basics/amd">AMD</a></td><td></td><td><a href="/docs/basics/amd">AMD</a></td></tr><tr><td><a href="/docs/get-started/install/updating">Updating</a></td><td></td><td><a href="/docs/get-started/install/updating">Updating</a></td></tr><tr><td><a href="/docs/get-started/install/intel">Intel</a></td><td></td><td></td></tr><tr><td><a href="/docs/get-started/install/conda-install">Conda</a></td><td></td><td><a href="/docs/get-started/install/conda-install">Conda</a></td></tr><tr><td><a href="/docs/get-started/install/vs-code">VS Code</a></td><td></td><td></td></tr><tr><td><a href="/docs/get-started/install/google-colab">Google Colab</a></td><td></td><td><a href="/docs/get-started/install/google-colab">Google Colab</a></td></tr></tbody></table>


# Install Unsloth via pip and uv

Install Unsloth Studio manually or install Unsloth Core with pip, uv, or a virtual environment.

**Unsloth Desktop**, **Unsloth Studio**, and **Unsloth Core** are installed differently. This page covers manual Studio installation and the code-based Core package.

### Unsloth Desktop

For the native desktop app, use the OS download instead of the commands below:

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

## **Unsloth Studio**

**Unsloth Studio** is our web UI. Follow below for native installation:

{% tabs %}
{% tab title="MacOS, Linux, WSL" %}
**Install Unsloth:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use the same command to update.
{% endtab %}

{% tab title="Windows" %}
**Install Unsloth:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

Use the same command to update.
{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
**Install Unsloth:**

```bash
brew install --cask unsloth
```

Update Unsloth:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
**Install Unsloth:**

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}

#### Launch:

{% columns %}
{% column %}
Standard local launch:

```bash
unsloth studio
```

For details about install and uninstallation please visit the [Unsloth Studio Install](/docs/new/studio/install) section.
{% endcolumn %}

{% column %}
Launch Unsloth securely over HTTPS:

```bash
unsloth studio --secure
```

Use a [free Cloudflare](/docs/basics/how-to-serve-local-llms-anywhere-secure-remote-access-with-cloudflare-and-unsloth) tunnel on Windows, Mac, or Linux.
{% endcolumn %}
{% endcolumns %}

### **Install from Main Repo**

#### **macOS, Linux, WSL developer installs:**

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -H 0.0.0.0 -p 8888
```

#### **Windows PowerShell developer installs:**

```powershell
winget install -e --id Python.Python.3.13 --source winget
winget install --id=astral-sh.uv  -e --source winget
winget install --id Git.Git -e --source winget
git clone https://github.com/unslothai/unsloth
cd unsloth
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -H 0.0.0.0 -p 8888
```

### **Nightly Install**

#### **Nightly - MacOS, Linux, WSL:**

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
git checkout nightly
./install.sh --local
```

Then to launch every time:

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

#### **Nightly - Windows:**

Run in Windows Powershell:

```bash
winget install -e --id Python.Python.3.13 --source winget
winget install --id=astral-sh.uv  -e --source winget
winget install --id Git.Git -e --source winget
git clone https://github.com/unslothai/unsloth
cd unsloth
git checkout nightly
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
```

Then to launch every time:

<pre class="language-bash"><code class="lang-bash"><strong>unsloth studio -H 0.0.0.0 -p 8888
</strong></code></pre>

### **Use an Old Version**

#### **Unsloth Desktop**

To use an old version of Unsloth Desktop, it is very simple. Just look through [our GitHub releases](https://github.com/unslothai/unsloth/releases) and select the specific release you want to install in the 'Assets' area.

#### **Unsloth Studio**

To use an old version of Unsloth Studio, follow the below with your desired version and names:

**macOS, Linux, WSL:**

```shellscript
uv pip install \
  --python "$HOME/.unsloth/studio/unsloth_studio/bin/python" \
  --reinstall --no-deps "unsloth==VERSION"

"$HOME/.unsloth/studio/unsloth_studio/bin/unsloth" studio -p 8888
```

**Windows:**&#x20;

```shellscript
uv pip install `
  --python "$env:USERPROFILE\.unsloth\studio\unsloth_studio\Scripts\python.exe" `
  --reinstall --no-deps "unsloth==VERSION"

& "$env:USERPROFILE\.unsloth\studio\unsloth_studio\Scripts\unsloth.exe" studio -p 8888
```

### Uninstall

The recommended way to fully remove Unsloth Studio is the uninstall script for your OS. It stops any running servers, removes the app, CLI command, launcher data, shortcuts, and platform-specific entries (macOS `.app` bundle + Launch Services; Windows Start Menu + registry + PATH):

**macOS, WSL, Linux:**

```shellscript
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
```

**Windows (PowerShell):**

```ps1
irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex 
```

#### Manual uninstall

If you prefer to remove only specific parts:

**1. Remove app only** (keeps history, chats, checkpoints, and exports intact):

* **macOS, WSL, Linux:** `rm -rf ~/.unsloth/studio/unsloth_studio`
* **Windows (PowerShell):** `Remove-Item -Recurse -Force "$HOME\.unsloth\studio\unsloth_studio"`

**2. Remove Unsloth entirely** (keeps other Unsloth tools intact):

* **macOS, WSL, Linux:** `rm -rf ~/.unsloth/studio`
* **Windows (PowerShell):** `Remove-Item -Recurse -Force "$HOME\.unsloth\studio"`

**3. Remove everything Unsloth-related:**

* **macOS, WSL, Linux:** `rm -rf ~/.unsloth`
* **Windows (PowerShell):** `Remove-Item -Recurse -Force "$HOME\.unsloth"`

{% hint style="warning" %}
Note: Step 3 deletes everything in history, chats, model checkpoints, and exports. This cannot be undone.
{% endhint %}

**4. Remove shortcuts and symlinks:**

**macOS:**

```shellscript
rm -rf ~/Applications/Unsloth\ Studio.app ~/Desktop/Unsloth\ Studio
```

**Linux:**&#x20;

```shellscript
rm -f ~/.local/share/applications/unsloth-studio.desktop ~/Desktop/unsloth-studio.desktop
```

**WSL / Windows (PowerShell):**

```shellscript
Remove-Item -Force "$HOME\Desktop\Unsloth Studio.lnk"
Remove-Item -Force "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk"
```

**5. Remove the CLI command:**

* **macOS, Linux, WSL:** `rm -f ~/.local/bin/unsloth`
* **Windows (PowerShell):** The installer added the venv's `Scripts` directory to your User PATH. To remove it, open Settings → System → About → Advanced system settings → Environment Variables, find `Path` under User variables, and remove the entry pointing to `.unsloth\studio\...\Scripts`.

{% hint style="info" %}
Note: Steps 1-5 dont touch your downloaded HF model files. See Deleting cached HF model files below if you want to reclaim that space.
{% endhint %}

### **Deleting cached HF model files**

You can delete old model files either from the bin icon in model search or by removing the relevant cached model folder from the default Hugging Face cache directory. By default, Hugging Face uses `~/.cache/huggingface/hub/` on macOS/Linux/WSL and `C:\Users\<username>\.cache\huggingface\hub\` on Windows.

* **MacOS, Linux, WSL:** `~/.cache/huggingface/hub/`
* **Windows:** `%USERPROFILE%\.cache\huggingface\hub\`

If `HF_HUB_CACHE` or `HF_HOME` is set, use that location instead. On Linux and WSL, `XDG_CACHE_HOME` can also change the default cache root.

## **Unsloth Core**

**Install uv with the following command:**

**macOS, WSL, Linux:**

```shellscript
curl -LsSf https://astral.sh/uv/install.sh | sh
```

**Windows (PowerShell):**

```powershell
irm https://astral.sh/uv/install.ps1 | iex
```

**Install unsloth core with uv pip (recommended) for the latest pip release:**

```bash
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto
```

Or just pip:

```bash
pip install unsloth
```

To install **vLLM and Unsloth** together, do:

```bash
uv pip install unsloth vllm --torch-backend=auto
```

To install the **latest main branch** of Unsloth, do:

{% code overflow="wrap" %}

```bash
uv pip install unsloth --torch-backend=auto
pip uninstall unsloth unsloth_zoo -y && pip install --no-deps git+https://github.com/unslothai/unsloth_zoo.git && pip install --no-deps git+https://github.com/unslothai/unsloth.git
```

{% endcode %}

For **venv and virtual environments installs** to isolate your installation to not break system packages, and to reduce irreparable damage to your system, use venv:

{% code overflow="wrap" %}

```bash
apt install python3.10-venv python3.11-venv python3.12-venv python3.13-venv -y
python -m venv unsloth_env
source unsloth_env/bin/activate
pip install --upgrade pip && pip install uv
uv pip install unsloth --torch-backend=auto
```

{% endcode %}

If you're installing Unsloth in Jupyter, Colab, or other notebooks, be sure to prefix the command with `!`. This isn't necessary when using a terminal

{% hint style="info" %}
Python 3.13 is now supported!
{% endhint %}

### Uninstall Unsloth Core

If you're still encountering dependency issues with Unsloth, many users have resolved them by forcing uninstalling and reinstalling Unsloth:

{% code overflow="wrap" %}

```bash
pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth
pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth_zoo
```

{% endcode %}

### Advanced Pip Installation

{% hint style="warning" %}
Do **NOT** use this if you have [Conda](/docs/get-started/install/conda-install).
{% endhint %}

Pip is a bit more complex since there are dependency issues. The pip command is different for `torch 2.2,2.3,2.4,2.5` and CUDA versions.

For other torch versions, we support `torch211`, `torch212`, `torch220`, `torch230`, `torch240` and for CUDA versions, we support `cu118` and `cu121` and `cu124`. For Ampere devices (A100, H100, RTX3090) and above, use `cu118-ampere` or `cu121-ampere` or `cu124-ampere`.

For example, if you have `torch 2.4` and `CUDA 12.1`, use:

```bash
pip install --upgrade pip
pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git"
```

Another example, if you have `torch 2.5` and `CUDA 12.4`, use:

```bash
pip install --upgrade pip
pip install "unsloth[cu124-torch250] @ git+https://github.com/unslothai/unsloth.git"
```

And other examples:

```bash
pip install "unsloth[cu121-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu118-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu118-torch240] @ git+https://github.com/unslothai/unsloth.git"

pip install "unsloth[cu121-torch230] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu121-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"

pip install "unsloth[cu121-torch250] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu124-ampere-torch250] @ git+https://github.com/unslothai/unsloth.git"
```

Or, run the below in a terminal to get the **optimal** pip installation command:

```bash
wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python -
```

Or, run the below manually in a Python REPL:

{% code overflow="wrap" %}

```python
# Licensed under the Apache License, Version 2.0 (the "License")
try: import torch
except: raise ImportError('Install torch via `pip install torch`')
from packaging.version import Version as V
import re
v = V(re.match(r"[0-9\.]{3,}", torch.__version__).group(0))
cuda = str(torch.version.cuda)
is_ampere = torch.cuda.get_device_capability()[0] >= 8
USE_ABI = torch._C._GLIBCXX_USE_CXX11_ABI
if cuda not in ("11.8", "12.1", "12.4", "12.6", "12.8", "13.0"): raise RuntimeError(f"CUDA = {cuda} not supported!")
if   v <= V('2.1.0'): raise RuntimeError(f"Torch = {v} too old!")
elif v <= V('2.1.1'): x = 'cu{}{}-torch211'
elif v <= V('2.1.2'): x = 'cu{}{}-torch212'
elif v  < V('2.3.0'): x = 'cu{}{}-torch220'
elif v  < V('2.4.0'): x = 'cu{}{}-torch230'
elif v  < V('2.5.0'): x = 'cu{}{}-torch240'
elif v  < V('2.5.1'): x = 'cu{}{}-torch250'
elif v <= V('2.5.1'): x = 'cu{}{}-torch251'
elif v  < V('2.7.0'): x = 'cu{}{}-torch260'
elif v  < V('2.7.9'): x = 'cu{}{}-torch270'
elif v  < V('2.8.0'): x = 'cu{}{}-torch271'
elif v  < V('2.8.9'): x = 'cu{}{}-torch280'
elif v  < V('2.9.1'): x = 'cu{}{}-torch290'
elif v  < V('2.9.2'): x = 'cu{}{}-torch291'
else: raise RuntimeError(f"Torch = {v} too new!")
if v > V('2.6.9') and cuda not in ("11.8", "12.6", "12.8", "13.0"): raise RuntimeError(f"CUDA = {cuda} not supported!")
x = x.format(cuda.replace(".", ""), "-ampere" if False else "") # is_ampere is broken due to flash-attn
print(f'pip install --upgrade pip && pip install --no-deps git+https://github.com/unslothai/unsloth-zoo.git && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git" --no-build-isolation')
```

{% endcode %}


# Install Unsloth on MacOS

Install Unsloth Desktop, Unsloth Studio, or Unsloth Core on macOS.

To install **Unsloth Desktop** to run and train MLX or GGUF models on your Mac, follow the steps below.

{% stepper %}
{% step %}

### Download

<a href="https://unsloth.ai/download/mac" class="button secondary" data-icon="apple">Download Unsloth for Mac</a>
{% endstep %}

{% step %}

### Install

1. Open the Unsloth installer (`.dmg` file)
2. Drag Unsloth to **Applications**
3. Launch the app and wait for installation to complete
   {% endstep %}

{% step %}

### Choose a model

Open 'Select model' dropdown ontop or 'Model hub' tab, choose a model and a quantization that fits your device, then download it. Once it finishes, start chatting - no setup required.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNCEVgKLJI0goPqjgcg9B%2Fmodel%20hub%20models.png?alt=media&amp;token=533b5e3c-a901-4b33-963e-4a703cc9d5a6" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

### Unsloth is now ready

To start chatting, type a message and press Enter.

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), web search, [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, embedding, and more
* **Generate media:** Create and train images, video, TTS locally

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpAGvwjGD0iVMZKBoyu7m%2Fgreeennn.png?alt=media&amp;token=d17a5528-8375-444c-9aff-f9e9f7903bcd" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Install Unsloth Studio manually

The command below installs **Unsloth Studio**, the browser-based web UI. It does not install Unsloth Desktop or the code-based Unsloth Core package.

#### **Terminal:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use same command to update.

#### **Homebrew:**

{% tabs %}
{% tab title="Install" %}

```bash
brew install --cask unsloth
```

{% endtab %}

{% tab title="Update" %}

```bash
brew upgrade --cask unsloth
```

{% endtab %}
{% endtabs %}

### Launch

Every time you want to launch Unsloth again:

{% tabs %}
{% tab title="Local launch" %}

```bash
unsloth studio
```

{% endtab %}

{% tab title="Remote access" %}

```bash
unsloth studio --secure
```

{% endtab %}
{% endtabs %}

For detailed Unsloth Studio install instructions and requirements, [view our guide](/docs/new/studio/install).

## Uninstall

### Desktop uninstall

1. Close Unsloth if it's running
2. Open **Finder**
3. Go to **Applications**
4. Find Unsloth
5. Right-click → **Move to Trash**

#### Unsloth Studio manual uninstall

The recommended way to fully remove Unsloth Studio is the uninstall script for your OS. It stops any running servers, removes the app, CLI command, launcher data, shortcuts, and platform-specific entries (macOS `.app` bundle + Launch Services; Windows Start Menu + registry + PATH):

```shellscript
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
```

If you prefer to remove only specific parts:

**1. Remove app only** (keeps history, chats, checkpoints, and exports intact):

* `rm -rf ~/.unsloth/studio/unsloth_studio`

**2. Remove Unsloth entirely** (keeps other Unsloth tools intact):

* &#x20;`rm -rf ~/.unsloth/studio`

**3. Remove everything Unsloth-related:**

* `rm -rf ~/.unsloth`

{% hint style="warning" %}
Note: Step 3 deletes everything in history, chats, model checkpoints, and exports. This cannot be undone.
{% endhint %}

**4. Remove shortcuts and symlinks:**

```shellscript
rm -rf ~/Applications/Unsloth\ Studio.app ~/Desktop/Unsloth\ Studio
```

**5. Remove the CLI command:**

* `rm -f ~/.local/bin/unsloth`

{% hint style="info" %}
Note: Steps 1-5 dont touch your downloaded HF model files. See Deleting cached HF model files below if you want to reclaim that space.
{% endhint %}

### **Deleting model files**

You can delete old model files either from the bin icon in model search or by removing the relevant cached model folder from the Hugging Face cache directory.

The default cache location is:

```bash
~/.cache/huggingface/hub/
```

If `HF_HUB_CACHE` or `HF_HOME` is set, use that location instead. You can check with:

```bash
echo ${HF_HUB_CACHE:-${HF_HOME:-${XDG_CACHE_HOME:-$HOME/.cache}/huggingface}/hub}
```

To delete a specific model, remove its folder (e.g. `models--unsloth--Llama-3.1-8B-bnb-4bit`) from the cache directory. To clear all cached models:

```bash
rm -rf ~/.cache/huggingface/hub/
```


# Install Unsloth on Windows

Install Unsloth Desktop, Unsloth Studio, or Unsloth Core on Windows.

To install Unsloth Desktop on Windows, follow the steps below.

{% stepper %}
{% step %}

### Download

<a href="https://unsloth.ai/download/windows" class="button secondary" data-icon="windows">Download Unsloth for Windows</a>
{% endstep %}

{% step %}

### Install

1. Open the Unsloth installer (`.exe` file)
2. Follow the setup prompts
3. Launch the app and wait for installation to complete
   {% endstep %}

{% step %}

### Choose a model

Open Select model or Model hub, choose a model and quantization, then download it. Once it finishes, start chatting - no setup required.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNCEVgKLJI0goPqjgcg9B%2Fmodel%20hub%20models.png?alt=media&amp;token=533b5e3c-a901-4b33-963e-4a703cc9d5a6" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

### Unsloth is now ready

To start chatting, type a message and press Enter.

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), web search, [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, embedding, and more
* **Generate media:** Create and train images, video, TTS locally

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpAGvwjGD0iVMZKBoyu7m%2Fgreeennn.png?alt=media&amp;token=d17a5528-8375-444c-9aff-f9e9f7903bcd" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Install Unsloth manually

The sections below cover the web-based **Unsloth Studio** and the original code-based **Unsloth Core**.

For **Unsloth Core** on Windows, choose Conda, Docker, or WSL below. If PyTorch is already installed, `pip install unsloth` may be enough.

<a href="/docs/get-started/install/windows-installation#method-1-windows-via-conda" class="button secondary">Conda Tutorial</a><a href="/docs/get-started/install/windows-installation#method-2-docker" class="button secondary">Docker Tutorial</a><a href="/docs/get-started/install/windows-installation#method-3-wsl" class="button secondary">WSL Tutorial</a>

### Unsloth Studio

See below for command to install our web UI [Unsloth Studio](/docs/new/studio/install)

#### **Powershell**:

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

Use the same command to update.

#### **Winget**:

{% tabs %}
{% tab title="Install" %}

```powershell
winget install --id UnslothAI.Unsloth -e
```

{% endtab %}

{% tab title="Update" %}

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}

#### **Launch:**

{% tabs %}
{% tab title="Local launch" %}

```powershell
unsloth studio
```

{% endtab %}

{% tab title="Remote access" %}

```powershell
unsloth studio --secure
```

{% endtab %}
{% endtabs %}

For detailed Unsloth Studio install instructions and requirements, [view our guide](/docs/new/studio/install).

Below are installation instructions for the original **Unsloth Core**:

### Method #1 - Windows via Conda:

{% stepper %}
{% step %}
**Install Miniconda (or Anaconda)**

Download Anaconda [here](https://www.anaconda.com/download). Our suggestion is to use [Miniconda](https://www.anaconda.com/docs/getting-started/miniconda/install#quickstart-install-instructions). To use it, first enter Powershell - search "Windows Powershell" in Start:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FvCgJ3hTR5ChVmCR1ndAh%2Fimage.png?alt=media&amp;token=bcabe210-793f-40ae-944a-a349dddc8c35" alt="" width="375"><figcaption></figcaption></figure>

Then it'll open up Powershell:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fm7J0b8Qz5W2aGktt3KA9%2Fimage.png?alt=media&amp;token=f84327e4-408f-492c-a909-982ed458f393" alt="" width="375"><figcaption></figcaption></figure>

Then copy paste the below: CTRL+C, and paste it in Powershell CTRL+V:

{% code overflow="wrap" %}

```ps
Invoke-WebRequest -Uri "https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe" -OutFile ".\miniconda.exe"
Start-Process -FilePath ".\miniconda.exe" -ArgumentList "/S" -Wait
del .\miniconda.exe
```

{% endcode %}

Accept the warning and press "Paste anyway" and wait.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZCFxO1FrYGk7sV7AmCe8%2Fimage.png?alt=media&amp;token=f753dbdb-efa9-462c-875b-0a18509a10cf" alt="" width="375"><figcaption></figcaption></figure>

It's downloading the installer like below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F2TzsqlGZyUI4wBT0hXW9%2Fimage.png?alt=media&amp;token=a680690a-3179-4525-bf83-0163424b5ddc" alt="" width="375"><figcaption></figcaption></figure>

After installing, open up open **Anaconda Powershell Prompt** to use Miniconda via Start -> Search for it:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FWoQSFFfmB26WT6BdzrvJ%2Fimage.png?alt=media&amp;token=747c6c4e-f676-4927-abad-cb667e757309" alt="" width="375"><figcaption></figcaption></figure>

Then you'll see:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNTTeflwW9Gw7lxJdCUcy%2Fimage.png?alt=media&amp;token=77f093b7-9fd5-47f2-856d-18f238f5a95e" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Make conda environment**

```bash
conda create --name unsloth_env python==3.12 -y
conda activate unsloth_env
```

**You will see:**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FEFYV6IhOeXIDbzxYHzEJ%2Fimage.png?alt=media&amp;token=e3452b73-cfd2-4148-a735-cfe400369c17" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Check `nvidia-smi` to confirm you have a GPU, and look for the CUDA version**

After typing `nvidia-smi` in Powershell, you should see something like below. If you don't have `nvidia-smi` or the below fails to pop up, you need to reinstall [NVIDIA drivers](https://www.nvidia.com/en-us/drivers/).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1CMAzx7LX8LEc8GHy1I9%2Fimage.png?alt=media&amp;token=a0ad52e3-be17-4dc4-ae97-ba400a639098" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Install PyTorch**

When running `nvidia-smi` you will see at the top right corner: "CUDA Version: 13.0". Install PyTorch in PowerShell via. Change `130` to your CUDA version - ensure the [version exists](https://pytorch.org/) and matches your CUDA driver version.

{% code overflow="wrap" %}

```bash
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130
```

{% endcode %}

You will see:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fo4aNeIbILGYvfpjYa1X2%2Fimage.png?alt=media&amp;token=444d6907-04e0-4d5e-8de5-d8cdcaf85364" alt="" width="563"><figcaption></figcaption></figure>

Try running this in Python via `python` after PyTorch is installed:

{% code overflow="wrap" %}

```python
import torch
print(torch.cuda.is_available())
A = torch.ones((10, 10), device = "cuda")
B = torch.ones((10, 10), device = "cuda")
A @ B
```

{% endcode %}

You should see a matrix of 10s. Also verify True for the first.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJkf39Nyfgyt4QmTLGhOx%2Fimage.png?alt=media&amp;token=2d1a16a6-e524-461a-ac75-a24b6bda333f" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Install Unsloth (only if PyTorch works!)**

{% hint style="danger" %}
**Confirm PyTorch works fine and runs - if not PyTorch is broken and that means your Windows machine might need a re installation of CUDA drivers unfortunately.**
{% endhint %}

In Powershell (after exiting Python via `exit()` , do and wait:

```bash
pip install unsloth
```

{% endstep %}

{% step %}
**Verify Unsloth works**

Now use any script in [Unsloth Notebooks](/docs/get-started/unsloth-notebooks) (save to .py file), or use the below basic script:

{% code expandable="true" %}

```python
from unsloth import FastLanguageModel, FastModel
import torch
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
max_seq_length = 512
url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl"
dataset = load_dataset("json", data_files = {"train" : url}, split = "train")

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/gemma-3-270m-it",
    max_seq_length = max_seq_length, # Choose any for long context!
    load_in_4bit = True,  # 4-bit quantization. False = 16-bit LoRA.
    load_in_8bit = False, # 8-bit quantization
    load_in_16bit = False, # 16-bit LoRA
    full_finetuning = False, # Use for full fine-tuning.
    trust_remote_code = False, # Enable to support new models
    # token = "hf_...", # use one if using gated models
)

# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 3407,
    max_seq_length = max_seq_length,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)

trainer = SFTTrainer(
    model = model,
    train_dataset = dataset,
    tokenizer = tokenizer,
    args = SFTConfig(
        max_seq_length = max_seq_length,
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 60,
        logging_steps = 1,
        output_dir = "outputs",
        optim = "adamw_8bit",
        seed = 3407,
        dataset_num_proc = 1,
    ),
)
trainer.train()
```

{% endcode %}

You should see:

```bash
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!
==((====))==  Unsloth 2026.1.4: Fast Gemma3 patching. Transformers: 4.57.6.
   \\   /|    NVIDIA GeForce RTX 3060. Num GPUs = 1. Max memory: 12.0 GB. Platform: Windows.
O^O/ \_/ \    Torch: 2.10.0+cu130. CUDA: 8.6. CUDA Toolkit: 13.0. Triton: 3.6.0
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.34. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: Gemma3 does not support SDPA - switching to fast eager.
Unsloth: Making `model.base_model.model.model` require gradients
Unsloth: Tokenizing ["text"] (num_proc=1):   0%|                 | 0/210289 [00:00<?, ? examples/s]�  Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
```

And training:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FfJalYwpN7ffC88J2dsOQ%2Fimage.png?alt=media&amp;token=e2129773-9255-4983-b52e-51de5729a7ab" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Method #2 - Docker:

Docker might be the easiest way for Windows users to get started with Unsloth as there is no setup needed or dependency issues. [**`unsloth/unsloth`**](https://hub.docker.com/r/unsloth/unsloth) is Unsloth's only Docker image. For [Blackwell](/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) and 50-series GPUs, use this same image - no separate image needed.

For installation instructions, please follow our [Docker guide](/docs/blog/how-to-fine-tune-llms-with-unsloth-and-docker), otherwise here is a quickstart guide:

{% stepper %}
{% step %}
**Install Docker and NVIDIA Container Toolkit.**

Install Docker via [Linux](https://docs.docker.com/engine/install/) or [Desktop](https://docs.docker.com/desktop/) (other). Then install [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation):

```bash
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
sudo apt-get update && sudo apt-get install -y \
  nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
```

{% endstep %}

{% step %}
**Run the container.**

[**`unsloth/unsloth`**](https://hub.docker.com/r/unsloth/unsloth) is Unsloth's only Docker image.

```bash
docker run -d -e JUPYTER_PASSWORD="mypassword" \
  -p 8888:8888 -p 2222:22 \
  -v $(pwd)/work:/workspace/work \
  --gpus all \
  unsloth/unsloth
```

{% endstep %}

{% step %}
**Access Jupyter Lab**

Go to [http://localhost:8888](http://localhost:8888/) and open Unsloth. Access the `unsloth-notebooks` tabs to see Unsloth notebooks.
{% endstep %}

{% step %}
**Start training with Unsloth**

If you're new, follow our step-by-step [Fine-tuning Guide](/docs/get-started/fine-tuning-llms-guide), [RL Guide](/docs/get-started/reinforcement-learning-rl-guide) or just save/copy any of our premade [notebooks](/docs/get-started/unsloth-notebooks).
{% endstep %}

{% step %}
**Docker issues - GPU not discovered?**

Try doing WSL via [#method-2-wsl](#method-2-wsl "mention")
{% endstep %}
{% endstepper %}

### Method #3 - WSL:

{% stepper %}
{% step %}
**Install WSL**

Open up Command Prompt, the Terminal, and install Ubuntu. Set the password if asked.

```bash
wsl.exe --install Ubuntu-24.04
wsl.exe -d Ubuntu-24.04
```

{% endstep %}

{% step %} <mark style="color:$primary;background-color:orange;">**If you did NOT do (1), so you already installed WSL**</mark>**, enter WSL by typing `wsl` and ENTER in the command prompt**

```bash
wsl
```

{% endstep %}

{% step %}
**Install Python**

{% code overflow="wrap" %}

```bash
sudo apt update
sudo apt install python3 python3-full python3-pip python3-venv -y
```

{% endcode %}
{% endstep %}

{% step %}
**Install PyTorch**

{% code overflow="wrap" %}

```bash
pip install torch torchvision --force-reinstall --index-url https://download.pytorch.org/whl/cu130
```

{% endcode %}

If you encounter permission issues, use `–break-system-packages` so `pip install torch torchvision --force-reinstall --index-url https://download.pytorch.org/whl/cu130 –break-system-packages`
{% endstep %}

{% step %}
**Install Unsloth and Jupyter Notebook**

{% code overflow="wrap" %}

```bash
pip install unsloth jupyter
```

{% endcode %}

If you encounter permission issues, use `–-break-system-packages` so `pip install unsloth jupyter –-break-system-packages`
{% endstep %}

{% step %}
**Launch Unsloth via Jupyter Notebook**

{% code overflow="wrap" %}

```bash
jupyter notebook
```

{% endcode %}

Then open up our notebooks within [Unsloth Notebooks](/docs/get-started/unsloth-notebooks)and load them up! You can also go to Colab notebooks and download > download .ipynb and load them.

![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVbqNWsG2CCHKJJjrnU4s%2Funknown.png?alt=media\&token=854a6d0e-fc84-4e44-bf8e-4bf254801692)
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
If you're using GRPO or plan to use vLLM, currently vLLM does not support Windows directly but only via WSL or Linux.
{% endhint %}

### **Troubleshooting /** Advanced

For **advanced installation instructions** or if you see weird errors during installations:

1. Install `torch` and `triton`. Go to <https://pytorch.org> to install it. For example `pip install torch torchvision torchaudio triton`
2. Confirm if CUDA is installated correctly. Try `nvcc`. If that fails, you need to install `cudatoolkit` or CUDA drivers.
3. If using an Intel GPU, you will need to follow our [Intel Windows guide](/docs/get-started/install/intel#windows-only-runtime-configurations)
4. Install `xformers` manually. You can try installing `vllm` and seeing if `vllm` succeeds. Check if `xformers` succeeded with `python -m xformers.info` Go to <https://github.com/facebookresearch/xformers>. Another option is to install `flash-attn` for Ampere GPUs.
5. Double check that your versions of Python, CUDA, CUDNN, `torch`, `triton`, and `xformers` are compatible with one another. The [PyTorch Compatibility Matrix](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix) may be useful.
6. Finally, install `bitsandbytes` and check it with `python -m bitsandbytes`
7. If Unsloth is not detecting or using your GPU and if you are using our Docker container on Windows, your CUDA toolkit version `nvcc --version` should match the version of CUDA shown by nvidia-smi on the host GPU support for Docker containers on Windows is not automatic. [You need to follow Docker's guide](https://docs.docker.com/desktop/features/gpu/).

## Uninstall

#### Unsloth Desktop

1. Close Unsloth if it is running.
2. Open **Settings** in Windows.
3. Go to **Apps** → **Installed apps**.
4. Find **Unsloth**, select the three-dot menu, then choose **Uninstall**.
5. Follow the prompts to remove the app.

Uninstalling Unsloth Desktop does not remove downloaded models or your files.

#### Unsloth Studio (manual installation)

The recommended way to fully remove Unsloth Studio is the uninstall script for your OS. It stops any running servers, removes the app, CLI command, launcher data, shortcuts, and platform-specific entries (macOS `.app` bundle + Launch Services; Windows Start Menu + registry + PATH):

```ps1
irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex 
```

#### Manual uninstall

If you prefer to remove only specific parts:

**1. Remove app only** (keeps history, chats, checkpoints, and exports intact):

* `Remove-Item -Recurse -Force "$HOME\.unsloth\studio\unsloth_studio"`

**2. Remove Unsloth entirely** (keeps other Unsloth tools intact):

* `Remove-Item -Recurse -Force "$HOME\.unsloth\studio"`

**3. Remove everything Unsloth-related:**

* `Remove-Item -Recurse -Force "$HOME\.unsloth"`

{% hint style="warning" %}
Note: Step 3 deletes everything in history, chats, model checkpoints, and exports. This cannot be undone.
{% endhint %}

**4. Remove shortcuts and symlinks:**

```shellscript
Remove-Item -Force "$HOME\Desktop\Unsloth Studio.lnk"
Remove-Item -Force "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk"
```

**5. Remove the CLI command:**

* **Windows (PowerShell):** The installer added the venv's `Scripts` directory to your User PATH. To remove it, open Settings → System → About → Advanced system settings → Environment Variables, find `Path` under User variables, and remove the entry pointing to `.unsloth\studio\...\Scripts`.

{% hint style="info" %}
Note: Steps 1-5 dont touch your downloaded HF model files. See Deleting cached HF model files below if you want to reclaim that space.
{% endhint %}

### **Deleting cached HF model files**

You can delete old model files either from the bin icon in model search or by removing the relevant cached model folder from the default Hugging Face cache directory. By default, Hugging Face uses `~/.cache/huggingface/hub/` on macOS/Linux/WSL and `C:\Users\<username>\.cache\huggingface\hub\` on Windows.

* **Windows:** `%USERPROFILE%\.cache\huggingface\hub\`

If `HF_HUB_CACHE` or `HF_HOME` is set, use that location instead. On Linux and WSL, `XDG_CACHE_HOME` can also change the default cache root.


# Install Unsloth via Docker

Install Unsloth using our official Docker container

Learn how to run and train models with the Unsloth Docker container. No setup required - all dependencies are pre-installed. Just pull the image and start running and training models on your local NVIDIA or AMD GPUs.

| **NVIDIA** Docker: [**`unsloth/unsloth`**](https://hub.docker.com/r/unsloth/unsloth) | **AMD** Docker: [**`unsloth/unsloth-rocm`**](https://hub.docker.com/r/unsloth/unsloth-rocm) |
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |

{% hint style="success" %}
**NEW Sep 2026: Unsloth Docker container is now updated with Unsloth Studio and AMD.**

Unsloth shares the same cache as notebooks and scripts to avoid unnecessary re-downloads.
{% endhint %}

### Quickstart

This guide section applies to systems with NVIDIA GPUs. If you have [AMD see here](#amd-quickstart).

{% tabs %}
{% tab title="Linux / WSL" %}
To get Unsloth's Docker on NVIDIA GPUs, run the following in terminal:

```bash
docker run -d --name unsloth --gpus all --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -p 8000:8000 -p 8888:8888 \
  -e JUPYTER_PASSWORD="mypassword" \
  -v "$PWD":/workspace/host \
  -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
  -v unsloth-studio:/opt/unsloth-studio \
  unsloth/unsloth
```

* Run it from your project folder, because that folder (`$PWD`) becomes `/workspace/host`.
* What the flags do: exposing ports 8000 (Unsloth Studio) and 8888 (JupyterLab); the three `-v` flags keep your files, models and Unsloth Studio data.
* Requirement: NVIDIA driver 570.26 or newer.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fz1bQUO7kRRL6jrzDKsRO%2FiShot_2026-09-17_16.30.18.png?alt=media&amp;token=1a5a23ee-e384-4bc7-b5f5-4cb934989094" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Windows (PowerShell)" %}
Run the following in Powershell:

```powershell
docker run -d --name unsloth --gpus all --ipc=host `
  --ulimit memlock=-1 --ulimit stack=67108864 `
  -p 8000:8000 -p 8888:8888 `
  -e JUPYTER_PASSWORD="mypassword" `
  -v "${PWD}:/workspace/host" `
  -v "${HOME}/.cache/huggingface:/workspace/.cache/huggingface" `
  -v "unsloth-studio:/opt/unsloth-studio" `
  unsloth/unsloth
```

* Run it from your project folder, because that folder (`${PWD}`) becomes `/workspace/host`.
* What the flags do: exposing ports 8000 (Unsloth Studio) and 8888 (JupyterLab); the three `-v` flags keep your files, models and Unsloth Studio data.
* Requirement: NVIDIA driver 570.26 or newer.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F70dKPYWYN9PrkPkMtbdg%2Fimage.png?alt=media&amp;token=5b4a4309-c97b-4e65-8daa-babb312a97ca" alt=""><figcaption></figcaption></figure>
{% endtab %}
{% endtabs %}

### Prerequisites

If you do not have Docker installed for NVIDIA Container Toolkit, follow the below:

{% tabs %}
{% tab title="Linux / WSL" %}
Install [Docker Engine](https://docs.docker.com/engine/install/) or [Docker Desktop](https://docs.docker.com/desktop/), if you haven't already:

```bash
curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
```

If you have an NVIDIA GPU, install [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation) if you haven't already:

```bash
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/docker/install_nvidia_toolkit.sh -o install_nvidia_toolkit.sh && sudo -E bash install_nvidia_toolkit.sh
```

{% endtab %}

{% tab title="Windows" %}
**Install Docker Desktop and the NVIDIA driver.**

Install [Docker Desktop for Windows](https://docs.docker.com/desktop/setup/install/windows-install/) and the latest [NVIDIA Windows driver](https://www.nvidia.com/Download/index.aspx). \
Then update WSL in PowerShell:

```powershell
wsl --update
```

In Docker Desktop settings, select **General** and check that **Use the WSL 2 based engine** is enabled. Wait for the engine to start before continuing.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNSQSDRFKVGyePWp7zKZA%2Fimage.png?alt=media&amp;token=0516785a-3fc1-41bc-8690-b98df7085fbf" alt=""><figcaption></figcaption></figure>

No separate NVIDIA Container Toolkit installation is needed. [Windows GPU troubleshooting](https://docs.docker.com/desktop/features/gpu/).
{% endtab %}
{% endtabs %}

### AMD Quickstart

This guide section applies to systems with AMD GPUs. If you have [NVIDIA see here](#quickstart).

{% tabs %}
{% tab title="Linux" %}
To get Unsloth's Docker on AMD GPUs, run the following in terminal:

```bash
GPU_FLAGS="--device /dev/kfd"
[ -e /dev/dri ] && GPU_FLAGS="$GPU_FLAGS --device /dev/dri"
for g in video render; do
  gid=$(getent group "$g" | cut -d: -f3)
  [ -n "$gid" ] && GPU_FLAGS="$GPU_FLAGS --group-add $gid"
done
docker run -d --name unsloth \
  $GPU_FLAGS \
  --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -p 8000:8000 -p 8888:8888 \
  -e JUPYTER_PASSWORD="mypassword" \
  -v "$PWD":/workspace/host \
  -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
  -v unsloth-studio:/opt/unsloth-studio \
  unsloth/unsloth-rocm
```

* Run it from your project folder, because that folder (`$PWD`) becomes `/workspace/host`.
* What the flags do: `$GPU_FLAGS` passes the GPU device nodes and group ids; ports 8000 (Unsloth Studio) and 8888 (JupyterLab); the three `-v` flags keep your files, models and Unsloth Studio data.
* Requirement: a working `amdgpu` driver. Built against ROCm 7.2, covers RDNA1 and newer, plus CDNA.
  {% endtab %}

{% tab title="WSL" %}
In WSL, AMD GPUs are reached through WSL2's DXG bridge. Run this inside your WSL2 distro, not PowerShell, WSL2 has no `/dev/kfd` so the GPU is passed as `/dev/dxg` instead:

```bash
docker run -d --name unsloth --device /dev/dxg --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -e HSA_ENABLE_DXG_DETECTION=1 \
  -e LD_LIBRARY_PATH=/usr/lib/wsl/lib:/opt/rocm/lib \
  -p 8000:8000 -p 8888:8888 \
  -e JUPYTER_PASSWORD="mypassword" \
  -v /usr/lib/wsl/lib:/usr/lib/wsl/lib:ro \
  -v "$PWD":/workspace/host \
  -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
  -v unsloth-studio:/opt/unsloth-studio \
  unsloth/unsloth-rocm
```

* Run it from inside WSL2 (`wsl -d Ubuntu`), from your project folder, because that folder (`$PWD`) becomes `/workspace/host`.
* What the flags do: `--device /dev/dxg` and the `/usr/lib/wsl/lib` mount pass the GPU through WSL2; ports 8000 (Unsloth Studio) and 8888 (JupyterLab); the three `-v` flags keep your files, models and Unsloth Studio data.
* Requirement: the AMD Windows driver with WSL2 support on the host
  {% endtab %}

{% tab title="Windows (PowerShell)" %}
Run the following in PowerShell:

```powershell
docker run -d --name unsloth --device /dev/dxg --ipc=host `
  --ulimit memlock=-1 --ulimit stack=67108864 `
  -e HSA_ENABLE_DXG_DETECTION=1 `
  -e LD_LIBRARY_PATH=/usr/lib/wsl/lib:/opt/rocm/lib `
  -p 8000:8000 -p 8888:8888 `
  -e JUPYTER_PASSWORD="mypassword" `
  -v "/usr/lib/wsl/lib:/usr/lib/wsl/lib:ro" `
  -v "${PWD}:/workspace/host" `
  -v "${HOME}/.cache/huggingface:/workspace/.cache/huggingface" `
  -v "unsloth-studio:/opt/unsloth-studio" `
  unsloth/unsloth-rocm
```

* Run it from your project folder, because that folder (`${PWD}`) becomes `/workspace/host`.
* Requirement: an AMD Windows driver with WSL2 support, and Docker Desktop using the WSL 2 engine.
  {% endtab %}
  {% endtabs %}

If you do not have Docker installed, run the following command:

```bash
curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
```

AMD needs no container toolkit, only the `amdgpu` driver on the host. On Windows, install Docker Desktop and the AMD driver with WSL2 support.

### 📖 Usage Guides

#### Unsloth Studio Setup

Unsloth Studio comes pre-installed, so you can chat with models, fine-tune them and generate images from the same Docker container.

{% stepper %}
{% step %}
**Start the Unsloth container.** \
Run the full `docker run` command for your OS in **Quickstart** and allow about a minute for startup. Skip this if your container is already running.

```bash
docker start unsloth
```

{% endstep %}

{% step %}
**Find your passwords.** \
If you don’t already know your password, check the container logs:

```bash
docker logs -f unsloth
```

Look for **Unsloth container ready** for links and password details. Press **Ctrl+C** to stop following the logs; the container keeps running.
{% endstep %}

{% step %}
**Sign in**

Open [`http://localhost:8000`](http://localhost:8000) and sign in as `unsloth`. On first setup, use the generated password and choose a new one when prompted. Otherwise, use your existing password.
{% endstep %}

{% step %}
**Forgot your password?**

```bash
docker exec unsloth unsloth studio reset-password --username unsloth
```

This generates a new password, signs out existing sessions and revokes API keys. No restart is needed. These commands assume your container is named `unsloth`. If needed, run `docker ps` to find its name or ID.

{% hint style="info" %}
If Docker runs on another machine (like a cloud GPU or a server), `localhost` means your own computer. You must forward the ports before you open the links. Run `ssh -L 8000:localhost:8000 -L 8888:localhost:8888 user@your-server`. See the [Security Notes](#security-notes) section for details.
{% endhint %}
{% endstep %}
{% endstepper %}

#### **Chat with a model**

{% hint style="info" %}
Unsloth Studio and JupyterLab share the same GPU. A model loaded in Unsloth keeps its GPU memory until you unload it. If you run out of memory, unload the model in Unsloth or stop the notebook kernel.
{% endhint %}

{% stepper %}
{% step %}
**Choose and download a model**

Open **Select model** at the top of the page. Browse **Recommended**, or enter a model name and click **Search Hub** to search Hugging Face. You can also browse models in **Model hub**.

For GGUF models, choose a quantization that fits your available RAM and VRAM. Avoid variants marked **OOM**.

Wait for the model to finish downloading and loading before sending your first message. You can select downloaded models again from **On Device**.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FgyDczCQqKkgh7ap2PuNU%2Fimage.png?alt=media&amp;token=740ee435-e68f-4811-8bb6-aa093526f74f" alt="Unsloth Studio model selector with Hugging Face search and recommended GGUF models." width="375"><figcaption><p>Browse recommended models or search by name.</p></figcaption></figure>

{% hint style="info" %}
Downloaded models are stored in `/workspace/.cache/huggingface`. With the Quickstart volume mount, this cache is shared with notebooks and stays on your computer even if you remove the container.
{% endhint %}
{% endstep %}

{% step %}
**Send a message**

Once the model is ready, type a message and press **Enter**. Try:

> Explain how Docker containers differ from virtual machines.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FKfzGAT0DLWirgWvkfK1x%2Fimage.png?alt=media&amp;token=afff9dd2-06d4-4f7b-bba5-8a4835143090" alt=""><figcaption></figcaption></figure>

See the [Unsloth Studio Chat](/docs/new/studio/chat) guide for web search and more chat settings.
{% endstep %}
{% endstepper %}

#### **Train a model**

You can train models across text, [vision](https://unsloth.ai/docs/basics/vision-fine-tuning) and [audio](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning), with support for [embeddings](https://unsloth.ai/docs/basics/embedding-finetuning) and image diffusion too. All from the same Docker image.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F66Vvgt0l5iqAkAksIRYF%2Fimage.png?alt=media&amp;token=8fcb1ad6-2b53-4128-87fa-a47bcd1deab4" alt="" width="563"><figcaption></figcaption></figure>

**Choose a model and dataset**

Select **Train** and open **Configure**. Choose a model and training method, then select a Hugging Face dataset or upload your own.

**Start training**

Review **Parameters** in **Simple** or **Advanced** mode, then click **Start Training**. Track progress, loss and GPU usage in **Current Run**.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FD7qpbGJLY5YMv0E0QgrB%2Ftrainingrun_LLM-unsloth.gif?alt=media&amp;token=feb766a7-1a7b-4f47-b5f4-b38e3a0d1470" alt=""><figcaption></figcaption></figure>

**Try your trained model**

When training finishes, click **Compare in Chat**. Test the original and fine-tuned models with the same prompts to see how their responses differ.

See the [Studio training guide](https://unsloth.ai/docs/new/studio/start) for more detail, or use [Data Recipes](https://unsloth.ai/docs/new/studio/data-recipe) to prepare your dataset.

#### **Generate images**

{% stepper %}
{% step %}
**Open Images and choose a model**

Select **Images** in the sidebar. The **Create** workflow opens by default.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6WVNnM49nQoTBfv57RAs%2Fimage%20(529).png?alt=media&amp;token=ff61651a-4560-4473-9232-d06024cb216e" alt="Unsloth Studio Images page with Create selected in the sidebar and prompt controls." width="375"><figcaption><p>Open Images to create an image from a text prompt.</p></figcaption></figure>

Open **Select image model** and choose from **Recommended**, such as Z-Image-Turbo. For GGUF models, pick a quantization that fits your device. **TIGHT** may run slowly; **OOM** is unlikely to fit.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1gPBEymJifbpIE9lGfvd%2Fimage%20(530).png?alt=media&amp;token=6f1b3e41-2c5a-4c5a-9c75-f624c6f4ede6" alt="Image model selector showing Z-Image-Turbo-GGUF and available quantizations." width="375"><figcaption><p>Choose an image model and a quantization for your device.</p></figcaption></figure>
{% endstep %}

{% step %}
**Enter a prompt and generate**

Describe the image you want to create. For example:

> Top-down shot of a koi pond in a Japanese garden, dozens of orange, white and black koi swimming in tight formation, water clear enough to see the stone bottom, red maple leaves floating on the surface. Bright midday light, saturated colour, crisp reflections. Realistic photo, 50mm.

Leave the image settings at their defaults and click **Generate**.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fy0qucNcG7XQlDVvv3XZE%2Fimage%20(493).png?alt=media&amp;token=47888a19-5384-4ace-b5d7-f90cf4fb9b0f" alt="Unsloth Studio Create images page with Z-Image-Turbo loaded, a koi pond prompt, a generated image, and Recipe and Download controls." width="563"><figcaption><p>An example generation with Z-Image-Turbo.</p></figcaption></figure>

Open your image in the gallery and click **Download** to save it, or **Recipe** to reuse its prompt and settings.

**Transform**, **Inpaint** and **Edit** depend on the loaded model. See the [image diffusion guide](https://unsloth.ai/docs/basics/diffusion-image) for supported workflows and settings.
{% endstep %}
{% endstepper %}

#### **Access JupyterLab**

{% columns %}
{% column %}
The Unsloth Docker image includes JupyterLab and ready-to-use notebooks.

Open [`http://localhost:8888`](http://localhost:8888) and sign in. Use the `JUPYTER_PASSWORD` you chose when starting the container. If you did not set one, use the generated password printed in the container log.

{% hint style="info" %}
JupyterLab passwords are set when the container is created. To change it, remove and recreate the container with a new password.
{% endhint %}
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQapcJDBCYgxaEfG5zEEn%2Fimage.png?alt=media&amp;token=8eb72fbd-8ca1-43f6-9c6a-e45cb7a06da0" alt="" width="375"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

After signing in, you’ll see **Unsloth Notebooks**, with folders grouped by model and task.

These folders contain shortcuts to the notebook files in `/workspace/unsloth-notebooks`. Each time the container starts, the notebooks are refreshed from GitHub without overwriting
\
your edits.

Double-click **01 Main Notebooks** to browse examples, or choose a category such as vision, speech or reinforcement learning.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FXceIgyo451mqw1hchpXR%2Fimage.png?alt=media&amp;token=b32a2968-0e94-4eff-b6a5-ed9f9b617ca4" alt="" width="563"><figcaption></figcaption></figure></div>

#### **Fine-tune a model in a notebook**

Unload any model loaded in Unsloth Studio before you start training, so the notebook has the GPU memory it needs.

Open a notebook in **01 Main Notebooks** and follow its instructions. Press **Shift+Enter** to run each cell in order. The first run downloads the model into the Hugging Face cache.

For example, `Gemma3_(4B)-Vision.ipynb` guides you through fine-tuning Gemma 3 4B to turn images of equations into LaTeX.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FyLdWCorjvMlW0OPkE2v4%2Fimage.png?alt=media&amp;token=ca2e62b5-5857-4cbf-89f4-735030f1123b" alt="" width="563"><figcaption></figcaption></figure></div>

After training, test your model using the notebook's **Inference** section. It tests the trained model on an equation image. The equation is below, recreated at higher resolution here for readability.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FGJdMUUr0ig4NQL5TMk2I%2Fimage.png?alt=media&amp;token=fa1eb118-dcbd-4ad3-a8f4-48e47659c143" alt="" width="563"><figcaption></figcaption></figure></div>

The model generates LaTeX from the original dataset image. It prints the LaTeX result below the cell.

#### **Save and export your model**

Follow the notebook's Saving section to save your trained LoRA adapters or export your merged model.

A relative path like `"gemma_3_lora"` saves next to the notebook, inside `/workspace/Unsloth Notebooks/...`. That folder is inside the container, so it is deleted when you remove the container. To keep your results on your computer, save under `/workspace/host`.

* **Save LoRA adapters locally:** This cell runs by default. Change the path to `/workspace/host`.

  ```python
  model.save_pretrained("/workspace/host/gemma_3_lora")
  processor.save_pretrained("/workspace/host/gemma_3_lora")
  ```
* **Export merged 16-bit model:** In the export cell, change `if False:` to `if True:` for the option you want. Only pick one.

  ```python
  if True: model.save_pretrained_merged("/workspace/host/unsloth_finetune", processor)
  ```

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FERw6laeLKXj5mLtoWB8y%2Fimage.png?alt=media&amp;token=5d43449b-cc98-4448-93f5-79599d39148f" alt="" width="563"><figcaption></figcaption></figure></div>

* **Upload to Hugging Face:** You need a token with write access to upload. Replace the `YOUR_USERNAME` and `YOUR_HF_TOKEN` placeholders with your own details.

For more guidance, see our [Fine-tuning Guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide), [RL Guide](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide), and [notebook collection](https://github.com/unslothai/notebooks).

#### Stop, restart and update the container

Commands take the container name or id, not the image name `unsloth/unsloth`.

Shut down the container but keep it:

```bash
docker stop unsloth
```

Bring it back in the same state:

```bash
docker start unsloth
```

Find your container again, running or not:

```bash
docker ps -a
```

Stop and delete the container:

```bash
docker rm -f unsloth
```

Deleting the container does not delete your models, your host files (`/workspace/host`), or Unsloth Studio data (accounts, chats, and exports). A new container with the same `-v` flags continues where you left off, and your Unsloth password stays the same. Anything else written inside the container is lost.

To delete Unsloth Studio's data permanently, remove its volume:

```bash
docker volume rm unsloth-studio
```

To update to the latest Unsloth version, pull the new image:

```bash
#NVIDIA GPUS
docker pull unsloth/unsloth
#AMD GPUS
docker pull unsloth/unsloth-rocm
```

Then remove the old container and run the Quickstart command again with the same flags. Your models, files and Unsloth Studio data carry over.

```bash
docker rm -f unsloth
```

#### **Connect over SSH**

SSH is off unless you set `SSH_KEY` or `PUBLIC_KEY`. You connect as the `root` user with key-only authentication on port 22 inside the container.

Generate an SSH key:

```bash
ssh-keygen -t ed25519 -f ~/.ssh/container_key
```

The SSH key is applied when the container is created, so you must recreate the container to add it. Use this full command to map the SSH port to 2222 on your host, set your passwords, and mount your volumes.

{% tabs %}
{% tab title="NVIDIA" %}

```bash
docker run -d --name unsloth --gpus all --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -p 8000:8000 -p 8888:8888 -p 2222:22 \
  -e JUPYTER_PASSWORD="choose-a-jupyter-password" \
  -e UNSLOTH_STUDIO_PASSWORD="choose-a-studio-password" \
  -e SSH_KEY="$(cat ~/.ssh/container_key.pub)" \
  -v "$PWD":/workspace/host \
  -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
  -v unsloth-studio:/opt/unsloth-studio \
  unsloth/unsloth
```

{% endtab %}

{% tab title="AMD" %}

```bash
GPU_FLAGS="--device /dev/kfd"
[ -e /dev/dri ] && GPU_FLAGS="$GPU_FLAGS --device /dev/dri"
for g in video render; do
  gid=$(getent group "$g" | cut -d: -f3)
  [ -n "$gid" ] && GPU_FLAGS="$GPU_FLAGS --group-add $gid"
done
docker run -d --name unsloth $GPU_FLAGS --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -p 8000:8000 -p 8888:8888 -p 2222:22 \
  -e JUPYTER_PASSWORD="choose-a-jupyter-password" \
  -e UNSLOTH_STUDIO_PASSWORD="choose-a-studio-password" \
  -e SSH_KEY="$(cat ~/.ssh/container_key.pub)" \
  -v "$PWD":/workspace/host \
  -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
  -v unsloth-studio:/opt/unsloth-studio \
  unsloth/unsloth-rocm
```

{% endtab %}
{% endtabs %}

`UNSLOTH_STUDIO_PASSWORD` applies only the first time the `unsloth-studio` volume is used. See [Advanced Settings](#advanced-settings) below for all environment variables, volumes and ports.

Connect to the container:

```bash
ssh -i ~/.ssh/container_key -p 2222 root@localhost
```

#### 📂 Container Structure

* `/workspace/host/` — Your mounted work directory
* `/workspace/.cache/huggingface` — Model and dataset downloads
* `/workspace/.cache/triton` — Compiled Triton kernels
* `/workspace/unsloth-notebooks/` — Example fine-tuning notebooks, including your edits
* `/workspace/Unsloth Notebooks` — Example fine-tuning notebooks grouped by topic, rebuilt on each start
* `/opt/unsloth-studio` — Unsloth Studio's accounts, chats, outputs, exports and runs

### Troubleshooting

#### **Unsloth not detecting or using my GPU**

If the model is not using your GPU specifically for Docker, try:

Pulling the latest image manually:

```bash
 #NVIDIA GPUS
 docker pull unsloth/unsloth:latest
 #AMD GPUS
 docker pull unsloth/unsloth-rocm:latest
```

* Start the container with GPU access:
  * `docker run`: `--gpus all`
  * Docker Compose: `capabilities: [gpu]`
  * AMD: `--device /dev/kfd --device /dev/dri` plus the `--group-add` ids (the `$GPU_FLAGS` block in Quickstart)
* On Linux (NVIDIA), make sure the NVIDIA Container Toolkit is installed, and docker is restarted after installing it.
* On Windows (NVIDIA):
  * Run `docker exec unsloth nvidia-smi`. If it lists your GPU, the container can see it.
  * Follow [Docker's guide](< https://docs.docker.com/desktop/features/gpu/>)
* On AMD: run `docker exec unsloth rocm-smi`. If it lists your GPU, the container can see it. A permission error on `/dev/kfd` means the `--group-add` ids were missing.

#### Port is already allocated

Change the host side port, e.g. `-p 8001:8000`, then open [http://localhost:8001](http://localhost:8001/).

#### Unsloth Studio stopped responding after an hour

Unsloth Studio auto shuts down if the auto generated password is not changed. Run `docker restart unsloth` to start it again and change password.

#### Files in your project folder are owned by root

The container runs as root, so use `sudo chown`, or `core` with `--user`.

### ⚙️ Advanced Settings

#### Environment variables

Pass any of these to `docker run` with `-e NAME=value` or `--env-file`. These apply to both `unsloth/unsloth` and `unsloth/unsloth-rocm`.

| Variable                           | Description                                                                       | Default                                            |
| ---------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------- |
| `UNSLOTH_STUDIO_PASSWORD`          | Initial Unsloth Studio password for user `unsloth`                                | Unset: generated once and printed in `docker logs` |
| `JUPYTER_PASSWORD`                 | JupyterLab password                                                               | Unset: generated once and printed in `docker logs` |
| `JUPYTER_PORT`                     | JupyterLab port inside container                                                  | `8888`                                             |
| `UNSLOTH_STUDIO_PORT`              | Unsloth Studio port inside the container                                          | `8000`                                             |
| `UNSLOTH_STUDIO_BOOTSTRAP_TIMEOUT` | Seconds before Unsloth Studio shuts down if the generated password is not changed | `3600`                                             |
| `SSH_KEY` or `PUBLIC_KEY`          | SSH public key for `root` login                                                   | Unset: SSH off                                     |
| `UNSLOTH_ALLOW_CPU`                | Allow the container to start without a GPU                                        | `1`, Unset when a GPU is visible                   |
| `UNSLOTH_STUDIO_SECURE`            | Serve Unsloth Studio only through a public Cloudflare HTTPS link                  | `0`                                                |
| `UNSLOTH_STUDIO_CLOUDFLARE`        | Share Unsloth Studio through a public Cloudflare link                             | `0`                                                |
| `UNSLOTH_JUPYTER_CLOUDFLARE`       | Share JupyterLab through a public Cloudflare link                                 | `0`                                                |
| `UNSLOTH_SKIP_NOTEBOOK_REFRESH`    | Skip updating the example notebooks from GitHub                                   | `0`                                                |
| `UNSLOTH_SKIP_NOTEBOOK_SYNC`       | Skip setting up the example notebooks                                             | `0`                                                |
| `HF_TOKEN`                         | Hugging Face access token                                                         | Unset                                              |
| `WANDB_API_KEY`                    | Weights & Biases API key                                                          | Unset                                              |

#### Volume

{% hint style="warning" %}
**Important**: Use volume mounts to preserve your work between container runs.
{% endhint %}

Anything not on a volume is lost when the container is removed. Mount these with `-v <host path or volume name>:<container path>`.

| Container path                  | Description                                                      | Suggested mount                                                 |
| ------------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------- |
| `/workspace/host`               | Your project files                                               | Your project folder, e.g. `"$PWD"`                              |
| `/workspace/.cache/huggingface` | Downloaded models and datasets                                   | Your host Hugging Face cache, e.g. `"$HOME/.cache/huggingface"` |
| `/opt/unsloth-studio`           | Unsloth Studio accounts, chats, trained models, exports and runs | Named volume `unsloth-studio`                                   |
| `/workspace/unsloth-notebooks`  | Example notebooks, including your edits                          | Optional                                                        |
| `/workspace/.cache/triton`      | Compiled GPU kernels                                             | Optional, speeds up restarts                                    |

* `/workspace` is the working directory and the folder JupyterLab opens in.
* Use a named volume for `/opt/unsloth-studio`, not a host folder. Unsloth Studio needs symlinks there, and a Windows or macOS host folder may not allow them, which stops the container at start.
* To give Unsloth Studio or JupyterLab more of your files, mount more folders under `/workspace`, e.g. `-v /data/datasets:/workspace/datasets`, and refer to them by that container path.

#### Ports

| Port   | Service                          |
| ------ | -------------------------------- |
| `8000` | Unsloth Studio                   |
| `8888` | JupyterLab                       |
| `22`   | SSH (only when `SSH_KEY` is set) |

Map a container port to any free host port with `-p <host>:<container>`. For example, to enable SSH on host port `2222`:

{% tabs %}
{% tab title="NVIDIA" %}

```bash
docker run -d --name unsloth --gpus all \
  -p 8000:8000 -p 8888:8888 -p 2222:22 \
  -e SSH_KEY="$(cat ~/.ssh/id_ed25519.pub)" \
  -v "$PWD":/workspace/host \
  -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
  -v unsloth-studio:/opt/unsloth-studio \
  unsloth/unsloth
```

{% endtab %}

{% tab title="AMD" %}

```bash
GPU_FLAGS="--device /dev/kfd"
[ -e /dev/dri ] && GPU_FLAGS="$GPU_FLAGS --device /dev/dri"
for g in video render; do
  gid=$(getent group "$g" | cut -d: -f3)
  [ -n "$gid" ] && GPU_FLAGS="$GPU_FLAGS --group-add $gid"
done
docker run -d --name unsloth $GPU_FLAGS \
  -p 8000:8000 -p 8888:8888 -p 2222:22 \
  -e SSH_KEY="$(cat ~/.ssh/id_ed25519.pub)" \
  -v "$PWD":/workspace/host \
  -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
  -v unsloth-studio:/opt/unsloth-studio \
  unsloth/unsloth-rocm
```

{% endtab %}
{% endtabs %}

### **🔒 Security Notes**

* The container runs as root. Use `unsloth/unsloth:core` with `--user <uid>:<gid>` if you need mounted files owned by your host user.
* SSH is off unless `SSH_KEY` or `PUBLIC_KEY` is set. It is key-only, as root, on port 22.
* Published ports listen on every interface. On a cloud host, bind to `127.0.0.1` or use `-e UNSLOTH_STUDIO_SECURE=1`.
* JupyterLab is a full shell. Anyone who can log in to JupyterLab can run any command in the container. Unsloth Studio and JupyterLab serve plain HTTP, so do not expose them to the internet directly.
* Unsloth Studio tools can run code. Studio's server-side tools are on by default and can run commands inside the container. Only mount host folders you are comfortable giving the container access to.
* Environment variables are visible to Docker users. Values passed with `-e` or `--env-file`, including tokens and passwords, show up in `docker inspect`. Anyone with access to the Docker daemon can read them.


# Install Unsloth on Linux

Install Unsloth Desktop, Unsloth Studio, or Unsloth Core on Linux.

To install Unsloth Desktop on Linux, follow the steps below.

{% stepper %}
{% step %}

### Download

<a href="https://unsloth.ai/download/linux" class="button secondary" data-icon="linux">Download Unsloth for Linux</a>
{% endstep %}

{% step %}

### Install

1. Open the Unsloth installer (`.deb` file)
2. Select Install
3. Launch the app and wait for installation to complete
   {% endstep %}

{% step %}

### Choose a model

Open Select model or Model hub, choose a model and quantization, then download it. Once it finishes, start chatting - no setup required.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNCEVgKLJI0goPqjgcg9B%2Fmodel%20hub%20models.png?alt=media&amp;token=533b5e3c-a901-4b33-963e-4a703cc9d5a6" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

### Unsloth is now ready

To start chatting, type a message and press Enter.

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), web search, [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, embedding, and more
* **Generate media:** Create and train images, video, TTS locally

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpAGvwjGD0iVMZKBoyu7m%2Fgreeennn.png?alt=media&amp;token=d17a5528-8375-444c-9aff-f9e9f7903bcd" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Advanced installation

Use these options for developer builds, nightly Unsloth Studio, or code-based Unsloth Core.

#### Install Unsloth Studio from the main repo

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -H 0.0.0.0 -p 8888
```

#### Install nightly Unsloth Studio

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
git checkout nightly
./install.sh --local
unsloth studio -H 0.0.0.0 -p 8888
```

#### Install Unsloth Core with uv

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto
```

Or install the latest pip release directly:

```bash
pip install unsloth
```

To install vLLM with Unsloth:

```bash
uv pip install unsloth vllm --torch-backend=auto
```

For virtual environments, CUDA and Torch-specific commands, and advanced pip options:

[**View the advanced pip installation guide**](https://unsloth.ai/docs/get-started/install/pip-install)

### Install Unsloth Studio manually

The command below installs **Unsloth Studio**, the browser-based web UI. It does not install the native Desktop app.

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use the same command to update Unsloth Studio.

#### Launch Unsloth Studio

{% tabs %}
{% tab title="Local launch" %}

```bash
unsloth studio
```

{% endtab %}

{% tab title="Remote access" %}

```bash
unsloth studio --secure
```

{% endtab %}
{% endtabs %}

For detailed Unsloth Studio install instructions and requirements, [view the Unsloth Studio guide](https://unsloth.ai/docs/new/studio/install).

### Uninstall

#### Unsloth Desktop

**Ubuntu or Debian (`.deb`)**

1. Close Unsloth if it is running.
2. Remove the package:

```bash
sudo apt remove unsloth
```

**AppImage**

1. Close Unsloth if it is running.
2. Delete the downloaded `.AppImage` file.

Uninstalling Unsloth Desktop does not remove downloaded models or your files.

#### Unsloth Studio (manual installation)

Run the Linux uninstall script:

```bash
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
```

This removes Unsloth Studio, its CLI command, launcher data, and Linux desktop shortcuts. It does not remove downloaded Hugging Face models.

#### Delete downloaded model files

The default Hugging Face cache is:

```bash
~/.cache/huggingface/hub/
```

If `HF_HUB_CACHE`, `HF_HOME`, or `XDG_CACHE_HOME` is set, use that location instead.


# Updating Unsloth

Update Unsloth Desktop, Unsloth Studio, or Unsloth Core.

### Update Unsloth Desktop

Unsloth Desktop checks for updates automatically after startup. To check manually, open **Settings** → **General** and select **Check for updates**.

* **macOS and Windows:** Select **Update now**. Unsloth installs the update and relaunches the app.
* **Linux AppImage:** In-app updates are supported for the AppImage build.
* **Linux `.deb`:** Download the latest package from the [Linux download page](https://unsloth.ai/download/linux) and install it over the existing version. Debian package installs update manually.

Unsloth Desktop updates independently from manual Unsloth Studio and Unsloth Core installations.

### Update Unsloth Studio

You can use the same install commands to update

#### **MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

#### **Windows PowerShell:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

#### **Homebrew:**

```bash
brew update && brew upgrade --cask unsloth
```

### Updating Unsloth Core:

```bash
pip install --upgrade unsloth unsloth_zoo
```

#### Updating Unsloth Core without dependency updates:

<pre class="language-bash" data-overflow="wrap"><code class="lang-bash">pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth
<strong>pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth_zoo
</strong></code></pre>

### **Use an Old Version**

#### **Unsloth Desktop**

To use an old version of Unsloth Desktop, it is very simple. Just look through [our GitHub releases](https://github.com/unslothai/unsloth/releases) and select the specific release you want to install in the 'Assets' area.

#### **Unsloth Studio**

To use an old version of Unsloth Studio, follow the below with your desired version and names:

**macOS, Linux, WSL:**

```shellscript
uv pip install \
  --python "$HOME/.unsloth/studio/unsloth_studio/bin/python" \
  --reinstall --no-deps "unsloth==VERSION"

"$HOME/.unsloth/studio/unsloth_studio/bin/unsloth" studio -p 8888
```

**Windows:**

```shellscript
uv pip install `
  --python "$env:USERPROFILE\.unsloth\studio\unsloth_studio\Scripts\python.exe" `
  --reinstall --no-deps "unsloth==VERSION"

& "$env:USERPROFILE\.unsloth\studio\unsloth_studio\Scripts\unsloth.exe" studio -p 8888
```

#### **Unsloth Core**

<pre class="language-bash" data-overflow="wrap"><code class="lang-bash"><strong>pip install --force-reinstall --no-cache-dir --no-deps unsloth==2025.1.5
</strong></code></pre>

'2025.1.5' is one of the previous old versions of Unsloth. Change it to a specific release listed on our [Github here](https://github.com/unslothai/unsloth/releases).


# Fine-tuning LLMs on Intel GPUs with Unsloth

Learn how to train and fine-tune large language models on Intel GPUs.

You can now fine-tune LLMs on your local Intel device with Unsloth! Read our guide on exactly how to get started with training your own custom model.

Before you begin, make sure you have:

* **Intel GPU:** Data Center GPU Max Series, Arc Series, or Intel Ultra AIPC
* **OS:** Linux (Ubuntu 22.04+ recommended) or Windows 11 (recommended)
* **Windows only:** Install Intel oneAPI Base Toolkit 2025.2.1 (select version 2025.2.1)
* **Intel Graphics driver:** Latest recommended driver for Windows/Linux
* **Python:** 3.10+

{% hint style="info" %}
Installing Unsloth Studio on an Intel GPU? See [Intel GPUs in the Studio install guide](/docs/new/studio/install#intel-gpus) for when the installer picks XPU PyTorch by itself and how to pin it.
{% endhint %}

### Build Unsloth with Intel Support

{% stepper %}
{% step %}

#### Create a new conda environment (Optional)

```bash
conda create -n unsloth-xpu python==3.10
conda activate unsloth-xpu
```

{% endstep %}

{% step %}

#### Install Unsloth

```bash
git clone https://github.com/unslothai/unsloth.git
cd unsloth
pip install .[intel-gpu-torch290]
```

{% hint style="info" %}
Linux Only: Install [vLLM](/docs/basics/inference-and-deployment/vllm-guide) (Optional)\
You can also install vLLM for [inference](/docs/basics/inference-and-deployment) and [RL](/docs/get-started/reinforcement-learning-rl-guide). Please follow [vLLM's guide](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#intel-xpu).
{% endhint %}
{% endstep %}

{% step %}

#### Verify your environments

```python
import torch
print(f"PyTorch version: {torch.__version__}")
print(f"XPU available: {torch.xpu.is_available()}")
print(f"XPU device count: {torch.xpu.device_count()}")
print(f"XPU device name: {torch.xpu.get_device_name(0)}")
```

{% endstep %}

{% step %}

#### Start fine-tuning.

You can directly use our Unsloth [notebooks](/docs/get-started/unsloth-notebooks) or view our dedicated [fine-tuning](/docs/get-started/fine-tuning-llms-guide) or [reinforcement learning](/docs/get-started/reinforcement-learning-rl-guide) guides.
{% endstep %}
{% endstepper %}

### Windows Only - Runtime Configurations

In Command Prompt with Administrator privilege, enable long path support in the Windows registry:

```bash
powershell -Command "Set-ItemProperty -Path "HKLM:\\SYSTEM\\CurrentControlSet\\Control\\FileSystem" -Name "LongPathsEnabled" -Value 1
```

This command only needs to be set once on a single machine. It does not need to be configured before each run. Then:

1. Download level-zero-win-sdk-1.20.2.zip from [GitHub](https://github.com/oneapi-src/level-zero/releases/tag/v1.20.2)
2. Unzip the level-zero-win-sdk-1.20.2.zip
3. In Command Prompt, under conda environment unsloth-xpu:

```bash
call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" -
set ZE_PATH=path\to\the\unzipped\level-zero-win-sdk-1.20.2
```

### Example 1: QLoRA Fine-tuning with SFT

This example demonstrates how to fine-tune a Qwen3-32B model using 4-bit QLoRA on an Intel GPU. QLoRA significantly reduces memory requirements, making it possible to fine-tune large models on consumer-grade hardware.

{% code expandable="true" %}

```python
from unsloth import FastLanguageModel, FastModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
max_seq_length = 2048 # Supports RoPE Scaling internally, so choose any!
# Get LAION dataset
url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl"
dataset = load_dataset("json", data_files = {"train" :
url}, split = "train")

# 4bit pre quantized models we support for fast downloading + no OOMs.
fourbit_models = [
"unsloth/Qwen3-32B-bnb-4bit",
"unsloth/Qwen3-14B-bnb-4bit",
"unsloth/Qwen3-8B-bnb-4bit",
"unsloth/Qwen3-4B-bnb-4bit",
"unsloth/Qwen3-1.7B-bnb-4bit",
"unsloth/Qwen3-0.6B-bnb-4bit",
# "unsloth/Qwen2.5-32B-bnb-4bit",
# "unsloth/Qwen2.5-14B-bnb-4bit",
# "unsloth/Qwen2.5-7B-bnb-4bit",
# "unsloth/Qwen2.5-3B-bnb-4bit",
# "unsloth/Qwen2.5-1.5B-bnb-4bit",
# "unsloth/Qwen2.5-0.5B-bnb-4bit",
# "unsloth/Llama-3.2-3B-bnb-4bit",
# "unsloth/Llama-3.2-1B-bnb-4bit",
# "unsloth/Llama-3.1-8B-bnb-4bit",
# "unsloth/Llama-3.1-70B-bnb-4bit",
# "unsloth/mistral-7b-bnb-4bit",
# "unsloth/Phi-4",
# "unsloth/Phi-3.5-mini-instruct",
# "unsloth/Phi-3-medium-4k-instruct",
# "unsloth/Phi-3-mini-4k-instruct",
# "unsloth/gemma-2-9b-bnb-4bit",
# "unsloth/gemma-2-27b-bnb-4bit",
] # More models at https://huggingface.co/unsloth

model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/Qwen3-32B-bnb-4bit",
max_seq_length = max_seq_length,
load_in_4bit = True,
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

model = FastLanguageModel.get_peft_model(
model,
r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_alpha = 16,
lora_dropout = 0, # Supports any, but = 0 is optimized
bias = "none", # Supports any, but = "none" is optimized
use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
random_state = 3407,
use_rslora = False, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
)

trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 1, # Recommended on Windows
packing = False, # Can make training 5x faster for short sequences.
args = SFTConfig(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4,
warmup_steps = 5,
max_steps = 60,
learning_rate = 2e-4,
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
dataset_num_proc=1, # Recommended on Windows
),
)

trainer.train()
```

{% endcode %}

### Example 2: Reinforcement Learning GRPO

GRPO is a [reinforcement learning](/docs/get-started/reinforcement-learning-rl-guide) technique for aligning language models with human preferences. This example shows how to train a model to follow a specific XML output format using multiple reward functions.

#### What is GRPO?

GRPO improves upon traditional RLHF by:

* Using group-based normalization for more stable training
* Supporting multiple reward functions for multi-objective optimization
* Being more memory efficient than PPO

{% code expandable="true" %}

```python
from unsloth import FastLanguageModel
import re
from trl import GRPOConfig, GRPOTrainer
from datasets import load_dataset, Dataset

max_seq_length = 1024  # Can increase for longer reasoning traces
lora_rank = 32  # Larger rank = smarter, but slower
max_prompt_length = 256

# Load and prep dataset
SYSTEM_PROMPT = """
Respond in the following format:
<reasoning>
...
</reasoning>
<answer>
...
</answer>
"""

XML_COT_FORMAT = """\
<reasoning>
{reasoning}
</reasoning>
<answer>
{answer}
</answer>
"""


def extract_xml_answer(text: str) -> str:
    answer = text.split("<answer>")[-1]
    answer = answer.split("</answer>")[0]
    return answer.strip()


def extract_hash_answer(text: str) -> str | None:
    if "####" not in text:
        return None
    return text.split("####")[1].strip()


# uncomment middle messages for 1-shot prompting
def get_gsm8k_questions(split: str = "train") -> Dataset:
    data = load_dataset("openai/gsm8k", "main")[split]  # type: ignore
    data = data.map(
        lambda x: {  # type: ignore
            "prompt": [
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": x["question"]},
            ],
            "answer": extract_hash_answer(x["answer"]),
        }
    )  # type: ignore
    return data  # type: ignore


# Reward functions
def correctness_reward_func(prompts, completions, answer, **kwargs) -> list[float]:
    responses = [completion[0]["content"] for completion in completions]
    q = prompts[0][-1]["content"]
    extracted_responses = [extract_xml_answer(r) for r in responses]
    print(
        "-" * 20,
        f"Question:\n{q}",
        f"\nAnswer:\n{answer[0]}",
        f"\nResponse:\n{responses[0]}",
        f"\nExtracted:\n{extracted_responses[0]}",
    )
    return [2.0 if r == a else 0.0 for r, a in zip(extracted_responses, answer)]


def int_reward_func(completions, **kwargs) -> list[float]:
    responses = [completion[0]["content"] for completion in completions]
    extracted_responses = [extract_xml_answer(r) for r in responses]
    return [0.5 if r.isdigit() else 0.0 for r in extracted_responses]


def strict_format_reward_func(completions, **kwargs) -> list[float]:
    """Reward function that checks if the completion has a specific format."""
    pattern = r"^<reasoning>\n.*?\n</reasoning>\n<answer>\n.*?\n</answer>\n$"
    responses = [completion[0]["content"] for completion in completions]
    matches = [re.match(pattern, r) for r in responses]
    return [0.5 if match else 0.0 for match in matches]


def soft_format_reward_func(completions, **kwargs) -> list[float]:
    """Reward function that checks if the completion has a specific format."""
    pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>"
    responses = [completion[0]["content"] for completion in completions]
    matches = [re.match(pattern, r) for r in responses]
    return [0.5 if match else 0.0 for match in matches]


def count_xml(text: str) -> float:
    count = 0.0
    if text.count("<reasoning>\n") == 1:
        count += 0.125
    if text.count("\n</reasoning>\n") == 1:
        count += 0.125
    if text.count("\n<answer>\n") == 1:
        count += 0.125
    count -= len(text.split("\n</answer>\n")[-1]) * 0.001
    if text.count("\n</answer>") == 1:
        count += 0.125
    count -= (len(text.split("\n</answer>")[-1]) - 1) * 0.001
    return count


def xmlcount_reward_func(completions, **kwargs) -> list[float]:
    contents = [completion[0]["content"] for completion in completions]
    return [count_xml(c) for c in contents]


if __name__ == "__main__":
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name="unsloth/Qwen3-0.6B",
        max_seq_length=max_seq_length,
        load_in_4bit=False,  # False for LoRA 16bit
        fast_inference=False,  # Enable vLLM fast inference
        max_lora_rank=lora_rank,
        gpu_memory_utilization=0.7,  # Reduce if out of memory
        device_map="xpu:0",
    )

    model = FastLanguageModel.get_peft_model(
        model,
        r=lora_rank,  # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
        target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj",
            "gate_proj",
            "up_proj",
            "down_proj",
        ],  # Remove QKVO if out of memory
        lora_alpha=lora_rank,
        use_gradient_checkpointing="unsloth",  # Enable long context finetuning
        random_state=3407,
    )

    dataset = get_gsm8k_questions()

    training_args = GRPOConfig(
        learning_rate=5e-6,
        adam_beta1=0.9,
        adam_beta2=0.99,
        weight_decay=0.1,
        warmup_ratio=0.1,
        lr_scheduler_type="cosine",
        optim="adamw_torch",
        logging_steps=1,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=1,  # Increase to 4 for smoother training
        num_generations=4,  # Decrease if out of memory
        max_prompt_length=max_prompt_length,
        max_completion_length=max_seq_length - max_prompt_length,
        # num_train_epochs=1,  # Set to 1 for a full training run
        max_steps=20,
        save_steps=250,
        max_grad_norm=0.1,
        report_to="none",  # Can use Weights & Biases
        output_dir="outputs",
    )

    trainer = GRPOTrainer(
        model=model,
        processing_class=tokenizer,
        reward_funcs=[
            xmlcount_reward_func,
            soft_format_reward_func,
            strict_format_reward_func,
            int_reward_func,
            correctness_reward_func,
        ],
        args=training_args,
        train_dataset=dataset,
        dataset_num_proc=1,  # Recommended on Windows
    )

    trainer.train()

```

{% endcode %}

## Troubleshooting

### Out of Memory (OOM) Errors

If you run out of memory, try these solutions:

1. **Reduce batch size:** Lower `per_device_train_batch_size`.
2. **Use a smaller model:** Start with a smaller model to reduce memory requirements.
3. **Reduce sequence length:** Lower `max_seq_length`.
4. **Reduce LoRA rank:** Use `r=8` instead of `r=16` or `r=32`.
5. **For GRPO, reduce number of generations:** Lower `num_generations`.

### (Windows Only) Intel Ultra AIPC iGPU Shared Memory

For Intel Ultra AIPC with recent GPU drivers on Windows, the shared GPU memory for the integrated GPU typically defaults to **57%** of system memory. For larger models (e.g., **Qwen3-32B**), or when using longer max sequence length, larger batch size, LoRA adapters with larger LoRA rank, etc., during fine-tuning, you could increase available VRAM by raising the percentage of system memory allocated to the iGPU.

You can adjust this by modifying the registry:

* Path: `Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\MemoryManager`
* Key to change:\
  `SystemPartitionCommitLimitPercentage` (set to a larger percentage)


# Fine-tuning LLMs on AMD GPUs with Unsloth Guide

Learn how to fine-tune large language models (LLMs) on AMD GPUs with Unsloth.

Fine-tune LLMs up to 2x faster with \~70% less memory on AMD hardware, no NVIDIA required. Unsloth supports AMD Radeon RDNA 3/3.5/4 (RX 6000–9000 series) on both Windows and Linux as well as data center GPUs including the MI300X (192GB).

{% stepper %}
{% step %}

#### **One-line installer**

To install Unsloth on AMD, the easiest way is to download our [Desktop app](/docs/desktop).

<a href="https://unsloth.ai/download/windows" class="button secondary" data-icon="windows">Download for Windows</a><a href="https://unsloth.ai/download/linux" class="button secondary" data-icon="linux">Download for Linux</a>\
\
For manual installation:

**Linux :**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows (PowerShell):**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

The manual steps below are for users who want to install the Unsloth Python library on AMD with the required dependencies.
{% endstep %}

{% step %}

#### **Make a new isolated environment (Optional)**

The steps below are for Linux. On Windows, use the Desktop app or the PowerShell installer above: they install AMD's Windows build of PyTorch for you.

To not break any system packages, make an isolated environment. Reminder to check what Python version you have! It might be `python3`, `python3.12`, `python3.13` etc. Python 3.10 to 3.13 works.

{% code overflow="wrap" %}

```bash
apt update && apt install python3.13-venv -y
python3.13 -m venv unsloth_env
source unsloth_env/bin/activate
pip install uv
```

{% endcode %}
{% endstep %}

{% step %}

#### **Install PyTorch**

PyPI only has the NVIDIA (CUDA) build of PyTorch, so install the ROCm build first from the PyTorch index that matches your ROCm version. Check it with `amd-smi version` (the `ROCm version:` line), then:

{% code overflow="wrap" %}

```bash
uv pip install "torch<2.15" torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/rocm7.2 --upgrade --force-reinstall
```

{% endcode %}

Change `rocm7.2` to your ROCm version. Available index tags are `rocm6.0` to `rocm6.4` and `rocm7.0` to `rocm7.2`. ROCm 6.5 to 6.9 uses `rocm6.4`, and ROCm 7.3 or newer uses `rocm7.2`. ROCm 6.0 or newer is required.

Or let this command pick the tag for you:

{% code overflow="wrap" %}

```bash
ROCM_VER="$(amd-smi version 2>/dev/null | sed -n 's/.*ROCm version: \([0-9]*\.[0-9]*\).*/\1/p' | head -n 1)"
[ -n "$ROCM_VER" ] || ROCM_VER="$(cut -d. -f1,2 /opt/rocm/.info/version 2>/dev/null)"
case "$ROCM_VER" in
    6.[0-4]|7.[0-2]) ROCM_TAG="rocm$ROCM_VER" ;;
    6.*) ROCM_TAG="rocm6.4" ;;
    *) ROCM_TAG="rocm7.2" ;;
esac
echo "ROCm ${ROCM_VER:-unknown}: using $ROCM_TAG"
uv pip install "torch<2.15" torchvision torchaudio \
    --index-url "https://download.pytorch.org/whl/$ROCM_TAG" --upgrade --force-reinstall
```

{% endcode %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FTVWdHJi4m8thLmCwOocd%2Fimage.png?alt=media&amp;token=c3c2dc1d-a405-4998-96ec-8dc320e3e4ce" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

#### **Install Unsloth**

Unsloth on PyPI depends on `torch`, so a plain install can swap the ROCm PyTorch you just installed for the CUDA one from PyPI. To prevent that, pin your installed PyTorch and install with `uv pip` (not plain `pip`):

{% code overflow="wrap" %}

```bash
uv pip freeze | grep -E '^(torch|torchvision|torchaudio)==' > torch-pins.txt
echo "triton ; sys_platform == 'never'" >> torch-pins.txt
echo "xformers ; sys_platform == 'never'" >> torch-pins.txt
uv pip install "unsloth[amd]" --overrides torch-pins.txt
```

{% endcode %}

`torch-pins.txt` should list a `torch==...+rocm...` line. The last two lines keep PyPI's `triton` from being installed over the Triton that came with your PyTorch, and skip PyPI's `xformers`, which is built for NVIDIA GPUs only.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fe0HmhIq3WvgX4hMh5B8N%2Fimage.png?alt=media&amp;token=8376914f-e8b8-4a84-bee9-83c80ed5ad92" alt="" width="547"><figcaption></figcaption></figure></div>

**Updating Unsloth**

Upgrade only Unsloth, with the same pins:

{% code overflow="wrap" %}

```bash
uv pip install --upgrade-package unsloth --upgrade-package unsloth-zoo "unsloth[amd]" --overrides torch-pins.txt
```

{% endcode %}

Do not run `uv pip install -U unsloth` or `--force-reinstall` without the pins: both resolve PyTorch again from PyPI and replace your ROCm build with the CUDA one. If that already happened, `import unsloth` reports an AMD GPU but no usable HIP accelerator: rerun the PyTorch step above, then this one.
{% endstep %}

{% step %}

#### **Start fine-tuning with Unsloth!**

And that's it. Try some examples in our [**Unsloth Notebooks**](/docs/get-started/unsloth-notebooks) page!

You can view our dedicated [fine-tuning](/docs/get-started/fine-tuning-llms-guide) or [reinforcement learning](/docs/get-started/reinforcement-learning-rl-guide) guides. Heres a brief example as well:

**1. Set environment variables**

{% code overflow="wrap" %}

```bash
export HF_HUB_DISABLE_XET=1            # Fixes HuggingFace download issues on AMD
```

{% endcode %}

**2. Load and configure model**

{% code overflow="wrap" %}

```python
from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/gemma-4-26b-a4b-it",
    max_seq_length = 2048,
    load_in_4bit = True,
)

model = FastModel.get_peft_model(
    model,
    r = 16,
    lora_alpha = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"],
)
```

{% endcode %}

**3. Train**

{% code overflow="wrap" %}

```python
from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    formatting_func = formatting_func,
    args = SFTConfig(
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        max_steps = 60,
        output_dir = "outputs",
        report_to = "none",
    ),
)

trainer_stats = trainer.train()
```

{% endcode %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwIcg0yj5i1hSyHDLyMYG%2Fimage.png?alt=media&amp;token=a6cbe0c7-fdbf-4feb-ad7b-7ac55ba91872" alt="" width="375"><figcaption></figcaption></figure></div>

***Note:** Flash Attention 2 is not installed on AMD GPUs by default, so Unsloth uses PyTorch's built-in SDPA attention instead. The warning can be safely ignored.*
{% endstep %}
{% endstepper %}

### :1234: Reinforcement Learning on AMD GPUs

You can use our :ledger:[gpt-oss RL auto win 2048](https://github.com/unslothai/notebooks/blob/main/nb/AMD-gpt_oss_\(20B\)_Reinforcement_Learning_2048_Game_BF16.ipynb) example on a MI300X (192GB) GPU. The goal is to play the 2048 game automatically and win it with RL. The LLM (gpt-oss 20b) auto devises a strategy to win the 2048 game, and we calculate a high reward for winning strategies, and low rewards for failing strategies.

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2bc5a2e25a51781fd945ab9e87e73821ed4eb6c9%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
The reward over time is increasing after around 300 steps or so!

The goal for RL is to maximize the average reward to win the 2048 game.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-8d7ea897fd57156a796e4f74aa2e3b60afe9d405%2F2048%20Auto%20Win%20Game%20Reward.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

We used an AMD MI300X machine (192GB) to run the 2048 RL example with Unsloth, and it worked well!

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-174890aa5f63632ebe6f3f212f1ced0d0e8dc381%2FScreenshot%202025-10-17%20052504.png?alt=media" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-f907ba596705496515fdfb39b49d649697317ca7%2FScreenshot%202025-10-17%20052641.png?alt=media" alt=""><figcaption></figcaption></figure></div>

You can also use our :ledger:[automatic kernel gen RL notebook](https://github.com/unslothai/notebooks/blob/main/nb/AMD-gpt_oss_\(20B\)_GRPO_BF16.ipynb) also with gpt-oss to auto create matrix multiplication kernels in Python. The notebook also devices multiple methods to counteract reward hacking.

{% columns %}
{% column width="50%" %}
The prompt we used to auto create these kernels was:

{% code overflow="wrap" %}

````
Create a new fast matrix multiplication function using only native Python code.
You are given a list of list of numbers.
Output your new function in backticks using the format below:
```
python
def matmul(A, B):
    return ...
```
````

{% endcode %}
{% endcolumn %}

{% column width="50%" %}
The RL process learns for example how to apply the Strassen algorithm for faster matrix multiplication inside of Python.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-ddb993e5d2c986794ede1f2b0d08897469b78506%2Fimage%20(1)%20(1)%20(1)%20(1)%20(1)%20(1).png?alt=media" alt="" width="375"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### :books:AMD Free One-click notebooks

AMD provides one-click notebooks equipped with **free 192GB VRAM MI300X GPUs** through their Dev Cloud. Train large models completely for free (no signup or credit card required):

* [Qwen3 (32B)](https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(32B\)_A100-Reasoning-Conversational.ipynb)
* [Llama 3.3 (70B)](https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb/AMD-Llama3.3_\(70B\)_A100-Conversational.ipynb)
* [Qwen3 (14B)](https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb/AMD-Qwen3_\(14B\)-Reasoning-Conversational.ipynb)
* [Mistral v0.3 (7B)](https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb/AMD-Mistral_v0.3_\(7B\)-Alpaca.ipynb)
* [GPT OSS MXFP4 (20B)](https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb/AMD-GPT_OSS_MXFP4_\(20B\)-Inference.ipynb) - Inference
* [Gemma4 (E2B)](https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)_Reinforcement_Learning_Sudoku_Game.ipynb) - RL Sudoku
* Unsloth Studio

{% embed url="<https://oneclickamd.ai/github/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_BF16.ipynb>" %}

You can use any Unsloth notebook by prepending <https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb> in [Unsloth Notebooks](/docs/get-started/unsloth-notebooks) by changing the link from <https://github.com/unslothai/notebooks/blob/main/nb/AMD-gpt_oss_(20B)_Reinforcement_Learning_2048_Game_BF16.ipynb>\
to <https://amd-ai-academy.com/github/unslothai/notebooks/blob/main/nb/AMD-Gemma4_(E2B)_Reinforcement_Learning_Sudoku_Game.ipynb>

{% columns %}
{% column width="33.33333333333333%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F7NNi4jLKvmZoRnLel9Kg%2Fimage.png?alt=media&amp;token=0379eda9-569c-4614-afb5-ffec463a7676" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="66.66666666666667%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRfKS1GAW7BqL9lGNTcxh%2Fimage.png?alt=media&amp;token=3a8aeb01-62a7-4d55-89a9-98526052e305" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}


# AMD AI Reinforcement Learning Hackathon with Unsloth

​​Learn hands-on techniques for ​Reinforcement Learning for AI models with Unsloth from Daniel Han, the creator of Unsloth.

You can view Unsloth's GitHub repo here: <https://github.com/unslothai/unsloth>

Here is the link to our AMD fine-tuning notebooks:

{% embed url="<https://github.com/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_BF16.ipynb>" %}

<https://github.com/unslothai/notebooks/blob/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_BF16.ipynb>

{% code overflow="wrap" %}

```bash
wget 'https://raw.githubusercontent.com/unslothai/notebooks/refs/heads/main/nb/gpt_oss_(20B)_Reinforcement_Learning_2048_Game_BF16.ipynb'
```

{% endcode %}

If wanting to upgrade Unsloth / Unsloth Zoo:

{% code overflow="wrap" %}

```bash
uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm7.0 --upgrade --force-reinstall

pip uninstall unsloth unsloth_zoo -y && \
    pip install git+https://github.com/unslothai/unsloth-zoo git+https://github.com/unslothai/unsloth --no-deps --force-reinstall --no-cache-dir
```

{% endcode %}

For bitsandbytes:

```bash
pip install "unsloth[amd] @ git+https://github.com/unslothai/unsloth"
```

If you see:

{% code overflow="wrap" %}

```
error: Failed to install: bitsandbytes-1.33.7rc0-py3-none-manylinux_2_24_x86_64.whl (bitsandbytes==1.33.7rc0 (from https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-1.33.7.preview-py3-none-manylinux_2_24_x86_64.whl)) Caused by: Wheel version does not match filename (0.49.2.dev0 != 1.33.7rc0), which indicates a malformed wheel. If this is intentional, set UV_SKIP_WHEEL_FILENAME_CHECK=1.
```

{% endcode %}

Do NOT use UV\_SKIP\_WHEEL\_FILENAME\_CHECK, instead ONLY use `pip install "unsloth[amd] @ git+https://github.com/unslothai/unsloth"` (NOT uv) since uv destroys bitsandbytes. Maybe add a check to the PRs if possible to catch these.

For AMD installation instructions, you can view our guide here:

{% content-ref url="/pages/GUxDmG8LaAiQinraCXCr" %}
[AMD](/docs/get-started/install/amd)
{% endcontent-ref %}


# Conda Install

To install Unsloth locally on Conda, follow the steps below:

{% hint style="warning" %}
Only use Conda if you have it. If not, use [Pip](/docs/get-started/install/pip-install).
{% endhint %}

Select either `pytorch-cuda=11.8,12.1` for CUDA 11.8 or CUDA 12.1. We support `python=3.10,3.11,3.12`.

```bash
conda create --name unsloth_env python=3.11 -y
conda activate unsloth_env
pip install unsloth
```

If you're looking to install Conda in a Linux environment, [read here](https://docs.anaconda.com/miniconda/), or run the below:

{% code overflow="wrap" %}

```bash
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
```

{% endcode %}


# How to Fine-tune LLMs in VS Code with Unsloth & Colab GPUs

Guide to fine-tuning models directly in Visual Studio Code via Unsloth and Google Colab.

You can now fine-tune LLMs directly from Visual Studio Code (VSCode), locally or by using Google Colab's extension. In this guide, you’ll learn how to use the open-source training [repo: Unsloth](https://github.com/unslothai/unsloth), to connect any [fine-tuning notebook](/docs/get-started/unsloth-notebooks) in VS Code to a Colab runtime, so you can train on your local or free Colab GPU. You can also view our video tutorial [here](#video-tutorial).

{% stepper %}
{% step %}

### VS Code and Colab Tutorial:

To begin we will need to have:

* Installed [VS Code](https://code.visualstudio.com/). Git (for cloning the notebook repo) should be installed by default.
* A **Google account** (to authenticate with Colab)
* Recommended: **Jupyter** extension (most VS Code setups already have it)
  {% endstep %}

{% step %}

#### Install the Colab extension in VS Code

1. Open **Extensions** in VS Code (`Ctrl+Shift+X` / `Cmd+Shift+X`)
2. Search for **“Colab”** and install the **Google Colab** extension

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJ1f5BS4EK0ok5QLy6vz5%2Fcolab_img_1.png?alt=media&amp;token=1faa7aac-c016-4c31-90ba-ccad655244e1" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Open an Unsloth notebook

1. Clone the Unsloth [notebooks repository](https://github.com/unslothai/notebooks):

```bash
git clone https://github.com/unslothai/notebooks
cd notebooks/nb
```

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwRL8pvgtLRpz5rcmS6vy%2FScreenshot%202026-02-18%20at%206.29.16%E2%80%AFPM.png?alt=media&amp;token=7a72c2ec-6f94-4015-8355-27e76ba9b974" alt="" width="375"><figcaption></figcaption></figure>

2. Open your desired notebook. Unsloth supports most models including [embedding](/docs/basics/embedding-finetuning), [TTS](/docs/basics/text-to-speech-tts-fine-tuning). For example, we'll use Qwen3-4B RL: `nb/Qwen3_(4B)-GRPO.ipynb`

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FleZtIzyflSgi1o1QcZDU%2Fcolab_img_2.png?alt=media&amp;token=805ebe8b-cb50-49a4-83d4-0daf96ad21c1" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Select a kernel and choose Colab

In the notebook toolbar, click **Select Kernel**, then choose **Colab**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FleGtAKSVehrlspPGvDy2%2Fcolab_img_3.png?alt=media&amp;token=ba3d8f1d-9cbb-44ed-8e52-8d0e52280607" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Add a new Colab server

After choosing **Colab**, you’ll see a dropdown with server options.

1. Click **+ Add New Colab Server**
2. The first time, a browser window may open for Google authentication
   * Log in, grant access, then return to VS Code

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZcDsB2i0aUStIKEeFTvF%2Fcolab_img_4.png?alt=media&amp;token=a7c3642f-db35-4215-933f-99c17ebc92a2" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Select GPU and name the server

1. Set **Hardware accelerator** to **GPU**
2. Choose a GPU type (for example **T4**, if available)
3. Give the server a name (anything you like)

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLuK8oheihdCNn9MAZSlk%2Fcolab_img_5.png?alt=media&amp;token=e5ed396f-afe2-4c09-b1b8-120c7301793c" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="info" %}
Note: GPU availability depends on your Colab plan and current capacity. If you don’t see GPU options, see troubleshooting below.
{% endhint %}
{% endstep %}

{% step %}

#### Pick the Python kernel

Once connected to the Colab server, select the **Python** kernel that appears for that runtime (usually a Python 3 kernel).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBexUiXJ7LRFBXs95tX2S%2Fcolab_img_6.png?alt=media&amp;token=391e343d-d2fa-4a77-8e3f-ef244adfcb4c" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run the notebook

* Click **Run All** in the notebook toolbar (or run cells top-to-bottom)
* Watch the setup cells install dependencies and then start the Unsloth workflow
* You can view our dedicated [fine-tuning](/docs/get-started/fine-tuning-llms-guide) or [reinforcement learning](/docs/get-started/reinforcement-learning-rl-guide) guides for more info on exactly how to get started with Unsloth.
  {% endstep %}
  {% endstepper %}

### Video Tutorial

{% embed url="<https://youtu.be/5NNggTrEWNQ>" %}

### Troubleshooting

#### After Colab server disconnects, the notebook won’t run on a new server

**What’s happening:**\
If the notebook stays open while the Colab server disconnects, VS Code can get stuck in a bad kernel/runtime state after reconnecting. Related [GitHub issue](https://github.com/googlecolab/colab-vscode/issues/200).

**Fix:** Close the notebook tab completely and open the notebook again.

#### You can’t select a GPU (only CPU shows up)

Possible causes and fixes:

* **Colab free tier capacity:** GPUs may be temporarily unavailable → try again later.
* **Not actually connected to a Colab runtime:** re-check **Select Kernel → Colab** and ensure a Colab server is active.
* **Account/region restrictions or limits reached:** you may need to wait or use a different Google account / plan.

#### Everything worked, but packages are “gone” after reconnecting

Colab runtimes are **ephemeral**. When a server restarts, you usually need to re-run the setup/install cells (often the first few cells in the notebook).


# Google Colab

To install and run Unsloth on Google Colab, follow the steps below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4d1b1778f3c8bde62a40130d7b4395b8bb1ce90f%2FColab%20Options.png?alt=media" alt=""><figcaption></figcaption></figure>

If you have never used a Colab notebook, a quick primer on the notebook itself:

1. **Play Button at each "cell".** Click on this to run that cell's code. You must not skip any cells and you must run every cell in chronological order. If you encounter errors, simply rerun the cell you did not run. Another option is to click CTRL + ENTER if you don't want to click the play button.
2. **Runtime Button in the top toolbar.** You can also use this button and hit "Run all" to run the entire notebook in 1 go. This will skip all the customization steps, but is a good first try.
3. **Connect / Reconnect T4 button.** T4 is the free GPU Google is providing. It's quite powerful!

The first installation cell looks like below: Remember to click the PLAY button in the brackets \[ ]. We grab our open source Github package, and install some other packages.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-135cd796b01420cc4d5ce3ca243e9065154070a5%2Fimage%20(13)%20(1)%20(1).png?alt=media" alt=""><figcaption></figcaption></figure>

### Colab Example Code

Unsloth example code to fine-tune gpt-oss-20b:

```python
from unsloth import FastLanguageModel, FastModel
import torch
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
max_seq_length = 2048 # Supports RoPE Scaling internally, so choose any!
# Get LAION dataset
url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl"
dataset = load_dataset("json", data_files = {"train" : url}, split = "train")

# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [
    "unsloth/gpt-oss-20b-unsloth-bnb-4bit", #or choose any model

] # More models at https://huggingface.co/unsloth

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/gpt-oss-20b",
    max_seq_length = 2048, # Choose any for long context!
    load_in_4bit = True,  # 4-bit quantization. False = 16-bit LoRA.
    load_in_8bit = False, # 8-bit quantization
    load_in_16bit = False, # [NEW!] 16-bit LoRA
    full_finetuning = False, # Use for full fine-tuning.
    # token = "hf_...", # use one if using gated models
)

# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 3407,
    max_seq_length = max_seq_length,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)

trainer = SFTTrainer(
    model = model,
    train_dataset = dataset,
    tokenizer = tokenizer,
    args = SFTConfig(
        max_seq_length = max_seq_length,
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 60,
        logging_steps = 1,
        output_dir = "outputs",
        optim = "adamw_8bit",
        seed = 3407,
    ),
)
trainer.train()

# Go to https://docs.unsloth.ai for advanced tips like
# (1) Saving to GGUF / merging to 16bit for vLLM
# (2) Continued training from a saved LoRA adapter
# (3) Adding an evaluation loop / OOMs
# (4) Customized chat templates
```


# Unsloth Model Catalog

Unsloth LLMs directory for all Unsloth [Dynamic](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs) GGUF, 4-bit, NVFP4 models on Hugging Face.

<a href="#qwen-models" class="button secondary">Qwen</a><a href="/docs/get-started/unsloth-model-catalog#gemma-models" class="button secondary">Gemma</a><a href="/docs/get-started/unsloth-model-catalog#deepseek-models" class="button secondary">DeepSeek</a><a href="#llama-models" class="button secondary">Llama</a><a href="#mistral-models" class="button secondary">Mistral</a><a href="https://unsloth.ai/docs/get-started/unsloth-model-catalog#glm-models" class="button secondary">GLM</a>

**GGUFs** let you run models in tools like [**Unsloth Desktop**](/docs/desktop)✨ and llama.cpp.\
Use **Instruct (4-bit)** safetensors for inference or fine-tuning via Unsloth.

#### **New & recommended models:**

| Model                                                                                                   | Variant                                                     | GGUF                                                                                                                                                            | 4-bit                                                                                                                                                                                  |
| ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [**Qwen3.8**](/docs/models/qwen3.8)                                                                     | 27B                                                         | [link](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF)                                                                                                         | [NVFP4](https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4)                                                                                                                              |
|                                                                                                         | [Flash-Next](/docs/models/qwen3.8-next)                     | [link](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)                                                                                                  |                                                                                                                                                                                        |
| [**DeepSeek-V4.1**](/docs/models/deepseek-v4.1-flash)                                                   | Flash (new)                                                 | [link](#new-and-recommended-models)                                                                                                                             |                                                                                                                                                                                        |
| [**GLM-5.3**](/docs/models/glm-5.3-flash)                                                               | [Flash](/docs/models/glm-5.3-flash)                         | [link](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF)                                                                                                       |                                                                                                                                                                                        |
|                                                                                                         | [5.3](/docs/models/glm-5.3)                                 | [link](https://huggingface.co/unsloth/GLM-5.3-GGUF)                                                                                                             |                                                                                                                                                                                        |
| [**DeepSeek-V4**](/docs/models/deepseek-v4)                                                             | V4-Flash-Vision-Exp                                         | [link](https://huggingface.co/unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF)                                                                                        |                                                                                                                                                                                        |
|                                                                                                         | V4-Pro-0813                                                 | [link](https://huggingface.co/unsloth/DeepSeek-V4-Pro-0813-GGUF)                                                                                                |                                                                                                                                                                                        |
|                                                                                                         | Flash-0731                                                  | [link](/docs/new/changelog)                                                                                                                                     | [link](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731)                                                                                                                          |
| [**Muse Glimmer**](/docs/models/deepseek-v4)                                                            | 30B                                                         | [link](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)                                                                                                    | [link](https://huggingface.co/unsloth/Muse-Glimmer-30B-unsloth-bnb-4bit)                                                                                                               |
| [**NVIDIA Nemotron-3.5**](/docs/models/nemotron-3.5) **(new)**                                          | 30B Lightning                                               | [link](https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF)                                                                               |                                                                                                                                                                                        |
| **Qwen3.8**                                                                                             | 2.4T-A95B                                                   | [link](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF)                                                                                                   |                                                                                                                                                                                        |
| **Kimi**                                                                                                | [**K3**](/docs/models/kimi-k3)                              | [link](https://huggingface.co/unsloth/Kimi-K3-GGUF)                                                                                                             | [link](https://huggingface.co/unsloth/Kimi-K3)                                                                                                                                         |
| [**Qwen3.6**](/docs/models/qwen3.6)                                                                     | 27B                                                         | [link](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF) • [MTP](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)                                            | [NVFP4](https://huggingface.co/unsloth/Qwen3.6-27B-NVFP4)                                                                                                                              |
|                                                                                                         | 35B-A3B                                                     | [link](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) • [MTP](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)                                    | [NVFP4](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-NVFP4)                                                                                                                          |
| [**Gemma 4**](/docs/models/gemma-4)                                                                     | [QAT](/docs/models/gemma-4/qat)                             | [link](https://huggingface.co/collections/unsloth/gemma-4-qat)                                                                                                  | [link](https://huggingface.co/collections/unsloth/gemma-4-qat)                                                                                                                         |
|                                                                                                         | 12B                                                         | [link](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF)                                                                                                      | [NVFP4](https://huggingface.co/unsloth/gemma-4-12b-it-NVFP4)                                                                                                                           |
|                                                                                                         | 26B-A4B                                                     | [link](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF)                                                                                                  | [NVFP4](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-NVFP4)                                                                                                                       |
|                                                                                                         | 31B                                                         | [link](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF)                                                                                                      | [link](https://huggingface.co/unsloth/gemma-4-31B-it-unsloth-bnb-4bit) • [NVFP4](https://huggingface.co/unsloth/gemma-4-31B-it-NVFP4)                                                  |
|                                                                                                         | E4B                                                         | [link](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF)                                                                                                      | [link](https://huggingface.co/unsloth/gemma-4-E4B-it-unsloth-bnb-4bit) • [NVFP4](https://huggingface.co/unsloth/gemma-4-E4B-it-NVFP4)                                                  |
|                                                                                                         | E2B                                                         | [link](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF)                                                                                                      | [link](https://huggingface.co/unsloth/gemma-4-E2B-it-unsloth-bnb-4bit) • [NVFP4](https://huggingface.co/unsloth/gemma-4-E2B-it-NVFP4)                                                  |
| [**DiffusionGemma**](/docs/models/diffusiongemma)                                                       | 26B-A4B                                                     | [link](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF)                                                                                           | —                                                                                                                                                                                      |
| **Kimi**                                                                                                | [K2.7-Code](/docs/models/kimi-k2.7-code)                    | [link](https://huggingface.co/unsloth/Kimi-K2.7-Code-GGUF)                                                                                                      | —                                                                                                                                                                                      |
|                                                                                                         | [**K2.6**](/docs/models/kimi-k2.6)                          | [link](https://huggingface.co/unsloth/Kimi-K2.6-GGUF)                                                                                                           | —                                                                                                                                                                                      |
| [**NVIDIA Nemotron 3**](/docs/models/nemotron-3-nano-omni)                                              | Nano-Omni-30B-A3B                                           | [link](https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF)                                                                                             | —                                                                                                                                                                                      |
| [**Qwen3.5**](https://github.com/unslothai/docs/blob/main/models/qwen3.5)                               | 35B-A3B                                                     | [link](https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF)                                                                                                     | —                                                                                                                                                                                      |
|                                                                                                         | 27B                                                         | [link](https://huggingface.co/unsloth/Qwen3.5-27B-GGUF)                                                                                                         | —                                                                                                                                                                                      |
|                                                                                                         | 122B-A10B                                                   | [link](https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF)                                                                                                   | —                                                                                                                                                                                      |
|                                                                                                         | 0.8B                                                        | [link](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)                                                                                                        | —                                                                                                                                                                                      |
|                                                                                                         | 2B                                                          | [link](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF)                                                                                                          | —                                                                                                                                                                                      |
|                                                                                                         | 4B                                                          | [link](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF)                                                                                                          | —                                                                                                                                                                                      |
|                                                                                                         | 9B                                                          | [link](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)                                                                                                          | —                                                                                                                                                                                      |
|                                                                                                         | 397B-A17B                                                   | [link](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF)                                                                                                   | —                                                                                                                                                                                      |
| **Qwen3**                                                                                               | [Coder-Next](/docs/models/qwen3-coder-next)                 | [link](https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF)                                                                                                    | —                                                                                                                                                                                      |
| NVIDIA Nemotron 3                                                                                       | [Super-120B-A12B](/docs/models/nemotron-3/nemotron-3-super) | [link](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF)                                                                                   | [link](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4)                                                                                                         |
|                                                                                                         | [Nano-4B](/docs/models/nemotron-3)                          | [link](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF)                                                                                           | —                                                                                                                                                                                      |
| **GLM**                                                                                                 | [4.7-Flash](/docs/models/tutorials/glm-4.7-flash)           | [link](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF)                                                                                                       | —                                                                                                                                                                                      |
|                                                                                                         | [5](/docs/models/tutorials/glm-5)                           | [link](https://huggingface.co/unsloth/GLM-5-GGUF)                                                                                                               | —                                                                                                                                                                                      |
| **Kimi**                                                                                                | [K2.5](/docs/models/tutorials/kimi-k2.5)                    | [link](https://huggingface.co/unsloth/Kimi-K2.5-GGUF)                                                                                                           | —                                                                                                                                                                                      |
| [**gpt-oss**](/docs/models/gpt-oss-how-to-run-and-fine-tune)                                            | 120B                                                        | [link](https://huggingface.co/unsloth/gpt-oss-120b-GGUF)                                                                                                        | [link](https://huggingface.co/unsloth/gpt-oss-120b-unsloth-bnb-4bit)                                                                                                                   |
|                                                                                                         | 20B                                                         | [link](https://huggingface.co/unsloth/gpt-oss-20b-GGUF)                                                                                                         | [link](https://huggingface.co/unsloth/gpt-oss-20b-unsloth-bnb-4bit)                                                                                                                    |
| **MiniMax**                                                                                             | [M2.5](/docs/models/tutorials/minimax-m25)                  | [link](https://huggingface.co/unsloth/MiniMax-M2.5-GGUF)                                                                                                        | —                                                                                                                                                                                      |
| NVIDIA [Nemotron 3](/docs/models/nemotron-3)                                                            | 30B                                                         | [link](https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF)                                                                                             | —                                                                                                                                                                                      |
| [**Qwen-Image**](/docs/models/tutorials/qwen-image-2512)                                                | 2512                                                        | [link](https://huggingface.co/unsloth/Qwen-Image-2512-GGUF)                                                                                                     | —                                                                                                                                                                                      |
|                                                                                                         | Edit-2511                                                   | [link](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)                                                                                                | —                                                                                                                                                                                      |
| [**Ministral 3**](/docs/models/tutorials/ministral-3)                                                   | 3B                                                          | [Instruct](https://huggingface.co/unsloth/Ministral-3-3B-Instruct-2512-GGUF) • [Reasoning](https://huggingface.co/unsloth/Ministral-3-3B-Reasoning-2512-GGUF)   | [Instruct](https://huggingface.co/unsloth/Ministral-3-14B-Instruct-2512-unsloth-bnb-4bit) • [Reasoning](https://huggingface.co/unsloth/Ministral-3-3B-Reasoning-2512-GGUF)             |
|                                                                                                         | 8B                                                          | [Instruct](https://huggingface.co/unsloth/Ministral-3-8B-Instruct-2512-GGUF) • [Reasoning](https://huggingface.co/unsloth/Ministral-3-8B-Reasoning-2512-GGUF)   | [Instruct](https://huggingface.co/unsloth/Ministral-3-8B-Instruct-2512-unsloth-bnb-4bit) • [Reasoning](https://huggingface.co/unsloth/Ministral-3-8B-Reasoning-2512-unsloth-bnb-4bit)  |
|                                                                                                         | 14B                                                         | [Instruct](https://huggingface.co/unsloth/Ministral-3-14B-Instruct-2512-GGUF) • [Reasoning](https://huggingface.co/unsloth/Ministral-3-14B-Reasoning-2512-GGUF) | [Instruct](https://huggingface.co/unsloth/Ministral-3-3B-Instruct-2512-unsloth-bnb-4bit) • [Reasoning](https://huggingface.co/unsloth/Ministral-3-14B-Reasoning-2512-unsloth-bnb-4bit) |
| [**Devstral 2**](/docs/models/tutorials/devstral-2)                                                     | 24B                                                         | [link](https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF)                                                                                  | —                                                                                                                                                                                      |
|                                                                                                         | 123B                                                        | [link](https://huggingface.co/unsloth/Devstral-2-123B-Instruct-2512-GGUF)                                                                                       | —                                                                                                                                                                                      |
| **Mistral Large 3**                                                                                     | 675B                                                        | [link](https://huggingface.co/unsloth/Mistral-Large-3-675B-Instruct-2512-GGUF)                                                                                  | [link](https://huggingface.co/unsloth/Mistral-Large-3-675B-Instruct-2512-NVFP4)                                                                                                        |
| [**Qwen3-Next**](/docs/models/tutorials/qwen3-next)                                                     | 80B-A3B-Instruct                                            | [link](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF)                                                                                         | [link](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-bnb-4bit/)                                                                                                           |
|                                                                                                         | 80B-A3B-Thinking                                            | [link](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF)                                                                                         | —                                                                                                                                                                                      |
| [**Qwen3-VL**](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune) | 2B-Instruct                                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-GGUF)                                                                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-unsloth-bnb-4bit)                                                                                                           |
|                                                                                                         | 2B-Thinking                                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-GGUF)                                                                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-unsloth-bnb-4bit)                                                                                                           |
|                                                                                                         | 4B-Instruct                                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF)                                                                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-unsloth-bnb-4bit)                                                                                                           |
|                                                                                                         | 4B-Thinking                                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-GGUF)                                                                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-unsloth-bnb-4bit)                                                                                                           |
|                                                                                                         | 8B-Instruct                                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF)                                                                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit)                                                                                                           |
|                                                                                                         | 8B-Thinking                                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-GGUF)                                                                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-unsloth-bnb-4bit)                                                                                                           |
|                                                                                                         | 30B-A3B-Instruct                                            | [link](https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Instruct-GGUF)                                                                                           | —                                                                                                                                                                                      |
|                                                                                                         | 30B-A3B-Thinking                                            | [link](https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Thinking-GGUF)                                                                                           | —                                                                                                                                                                                      |
|                                                                                                         | 32B-Instruct                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-32B-Instruct-GGUF)                                                                                               | [link](https://huggingface.co/unsloth/Qwen3-VL-32B-Instruct-unsloth-bnb-4bit)                                                                                                          |
|                                                                                                         | 32B-Thinking                                                | [link](https://huggingface.co/unsloth/Qwen3-VL-32B-Thinking-GGUF)                                                                                               | [link](https://huggingface.co/unsloth/Qwen3-VL-32B-Thinking-unsloth-bnb-4bit)                                                                                                          |
|                                                                                                         | 235B-A22B-Instruct                                          | [link](https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Instruct-GGUF)                                                                                         | —                                                                                                                                                                                      |
|                                                                                                         | 235B-A22B-Thinking                                          | [link](https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Thinking-GGUF)                                                                                         | —                                                                                                                                                                                      |
| [**Qwen3-2507**](/docs/models/tutorials/qwen3-next)                                                     | 30B-A3B-Instruct                                            | [link](https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF)                                                                                         | —                                                                                                                                                                                      |
|                                                                                                         | 30B-A3B-Thinking                                            | [link](https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF)                                                                                         | —                                                                                                                                                                                      |
|                                                                                                         | 235B-A22B-Instruct                                          | [link](https://huggingface.co/unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF/)                                                                                      | —                                                                                                                                                                                      |
| [**Qwen3-Coder**](/docs/models/tutorials/qwen3-coder-how-to-run-locally)                                | 30B-A3B                                                     | [link](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF)                                                                                        | —                                                                                                                                                                                      |
| [**GLM**](/docs/models/tutorials/glm-4.6-how-to-run-locally)                                            | 4.7                                                         | [link](https://huggingface.co/unsloth/GLM-4.7-GGUF)                                                                                                             | —                                                                                                                                                                                      |
|                                                                                                         | 4.6V-Flash                                                  | [link](https://huggingface.co/unsloth/GLM-4.6V-Flash-GGUF)                                                                                                      | —                                                                                                                                                                                      |
| [**DeepSeek-V3.1**](/docs/models/tutorials/deepseek-v3.1-how-to-run-locally)                            | Terminus                                                    | [link](https://huggingface.co/unsloth/DeepSeek-V3.1-Terminus-GGUF)                                                                                              | —                                                                                                                                                                                      |
|                                                                                                         | V3.1                                                        | [link](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF)                                                                                                       | —                                                                                                                                                                                      |

#### **DeepSeek models:**

| Model             | Variant                | GGUF                                                                      | Instruct (4-bit)                                                                      |
| ----------------- | ---------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| **DeepSeek-V3.1** | Terminus               | [link](https://huggingface.co/unsloth/DeepSeek-V3.1-Terminus-GGUF)        |                                                                                       |
|                   | V3.1                   | [link](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF)                 |                                                                                       |
| **DeepSeek-V3**   | V3-0324                | [link](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF)              | —                                                                                     |
|                   | V3                     | [link](https://huggingface.co/unsloth/DeepSeek-V3-GGUF)                   | —                                                                                     |
| **DeepSeek-R1**   | R1-0528                | [link](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF)              | —                                                                                     |
|                   | R1-0528-Qwen3-8B       | [link](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF)     | [link](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-unsloth-bnb-4bit)     |
|                   | R1                     | [link](https://huggingface.co/unsloth/DeepSeek-R1-GGUF)                   | —                                                                                     |
|                   | R1 Zero                | [link](https://huggingface.co/unsloth/DeepSeek-R1-Zero-GGUF)              | —                                                                                     |
|                   | Distill Llama 3 8 B    | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF)  | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit)  |
|                   | Distill Llama 3.3 70 B | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF) | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-bnb-4bit)         |
|                   | Distill Qwen 2.5 1.5 B | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-unsloth-bnb-4bit) |
|                   | Distill Qwen 2.5 7 B   | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-7B-GGUF)   | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-7B-unsloth-bnb-4bit)   |
|                   | Distill Qwen 2.5 14 B  | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF)  | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-unsloth-bnb-4bit)  |
|                   | Distill Qwen 2.5 32 B  | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-32B-GGUF)  | [link](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit)          |

#### **Llama models:**

| Model                                        | Variant             | GGUF                                                                           | 4-bit                                                                                  |
| -------------------------------------------- | ------------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| [**Muse Glimmer**](/docs/models/deepseek-v4) | 30B                 | [link](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)                   | [link](https://huggingface.co/unsloth/Muse-Glimmer-30B-NVFP4)                          |
| **Llama 4**                                  | Scout 17 B-16 E     | [link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF)     | [link](https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-4bit) |
|                                              | Maverick 17 B-128 E | [link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF) | —                                                                                      |
| **Llama 3.3**                                | 70 B                | [link](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF)             | [link](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-bnb-4bit)                 |
| **Llama 3.2**                                | 1 B                 | [link](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-GGUF)              | [link](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-bnb-4bit)                  |
|                                              | 3 B                 | [link](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-GGUF)              | [link](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-bnb-4bit)                  |
|                                              | 11 B Vision         | —                                                                              | [link](https://huggingface.co/unsloth/Llama-3.2-11B-Vision-Instruct-unsloth-bnb-4bit)  |
|                                              | 90 B Vision         | —                                                                              | [link](https://huggingface.co/unsloth/Llama-3.2-90B-Vision-Instruct-bnb-4bit)          |
| **Llama 3.1**                                | 8 B                 | [link](https://huggingface.co/unsloth/Llama-3.1-8B-Instruct-GGUF)              | [link](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit)             |
|                                              | 70 B                | —                                                                              | [link](https://huggingface.co/unsloth/Meta-Llama-3.1-70B-Instruct-bnb-4bit)            |
|                                              | 405 B               | —                                                                              | [link](https://huggingface.co/unsloth/Meta-Llama-3.1-405B-Instruct-bnb-4bit)           |
| **Llama 3**                                  | 8 B                 | —                                                                              | [link](https://huggingface.co/unsloth/llama-3-8b-Instruct-bnb-4bit)                    |
|                                              | 70 B                | —                                                                              | [link](https://huggingface.co/unsloth/llama-3-70b-bnb-4bit)                            |
| **Llama 2**                                  | 7 B                 | —                                                                              | [link](https://huggingface.co/unsloth/llama-2-7b-chat-bnb-4bit)                        |
|                                              | 13 B                | —                                                                              | [link](https://huggingface.co/unsloth/llama-2-13b-bnb-4bit)                            |
| **CodeLlama**                                | 7 B                 | —                                                                              | [link](https://huggingface.co/unsloth/codellama-7b-bnb-4bit)                           |
|                                              | 13 B                | —                                                                              | [link](https://huggingface.co/unsloth/codellama-13b-bnb-4bit)                          |
|                                              | 34 B                | —                                                                              | [link](https://huggingface.co/unsloth/codellama-34b-bnb-4bit)                          |

#### **Gemma models:**

| Model             | Variant       | GGUF                                                              | Instruct (4-bit)                                                             |
| ----------------- | ------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **Gemma 4**       | E2B           | [link](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF)        | [link](https://huggingface.co/unsloth/gemma-4-E2B-it-unsloth-bnb-4bit)       |
|                   | E4B           | [link](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF)        | [link](https://huggingface.co/unsloth/gemma-4-E4B-it-unsloth-bnb-4bit)       |
|                   | 26B-A4B       | [link](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF)    | —                                                                            |
|                   | 31B           | [link](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF)        | [link](https://huggingface.co/unsloth/gemma-4-31B-it-unsloth-bnb-4bit)       |
| **FunctionGemma** | 270M          | [link](https://huggingface.co/unsloth/functiongemma-270m-it-GGUF) | —                                                                            |
| **Gemma 3n**      | E2B           | ​[link](https://huggingface.co/unsloth/gemma-3n-E2B-it-GGUF)      | [link](https://huggingface.co/unsloth/gemma-3n-E2B-it-unsloth-bnb-4bit)      |
|                   | E4B           | [link](https://huggingface.co/unsloth/gemma-3n-E4B-it-GGUF)       | [link](https://huggingface.co/unsloth/gemma-3n-E4B-it-unsloth-bnb-4bit)      |
| **Gemma 3**       | 270M          | [link](https://huggingface.co/unsloth/gemma-3-270m-it-GGUF)       | [link](https://huggingface.co/unsloth/gemma-3-270m-it)                       |
|                   | 1 B           | [link](https://huggingface.co/unsloth/gemma-3-1b-it-GGUF)         | [link](https://huggingface.co/unsloth/gemma-3-1b-it-unsloth-bnb-4bit)        |
|                   | 4 B           | [link](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF)         | [link](https://huggingface.co/unsloth/gemma-3-4b-it-unsloth-bnb-4bit)        |
|                   | 12 B          | [link](https://huggingface.co/unsloth/gemma-3-12b-it-GGUF)        | [link](https://huggingface.co/unsloth/gemma-3-12b-it-unsloth-bnb-4bit)       |
|                   | 27 B          | [link](https://huggingface.co/unsloth/gemma-3-27b-it-GGUF)        | [link](https://huggingface.co/unsloth/gemma-3-27b-it-unsloth-bnb-4bit)       |
| **MedGemma**      | 4 B (vision)  | [link](https://huggingface.co/unsloth/medgemma-4b-it-GGUF)        | [link](https://huggingface.co/unsloth/medgemma-4b-it-unsloth-bnb-4bit)       |
|                   | 27 B (vision) | [link](https://huggingface.co/unsloth/medgemma-27b-it-GGUF)       | [link](https://huggingface.co/unsloth/medgemma-27b-text-it-unsloth-bnb-4bit) |
| **Gemma 2**       | 2 B           | [link](https://huggingface.co/unsloth/gemma-2-it-GGUF)            | [link](https://huggingface.co/unsloth/gemma-2-2b-it-bnb-4bit)                |
|                   | 9 B           | —                                                                 | [link](https://huggingface.co/unsloth/gemma-2-9b-it-bnb-4bit)                |
|                   | 27 B          | —                                                                 | [link](https://huggingface.co/unsloth/gemma-2-27b-it-bnb-4bit)               |

#### **Qwen models:**

| Model                                                                                                   | Variant                                     | GGUF                                                                         | Instruct (4-bit)                                                                |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [**Qwen3.8**](/docs/models/qwen3.8) **(new)**                                                           | 27B                                         | [link](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF)                      | [NVFP4](https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4)                       |
|                                                                                                         | 2.4T-A95B                                   | [link](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF)                |                                                                                 |
| [**Qwen3.6**](/docs/models/qwen3.6)                                                                     | 27B                                         | [link](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF)                      | —                                                                               |
|                                                                                                         | 35B-A3B                                     | [link](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF)                  | —                                                                               |
| [**Qwen3.5**](https://github.com/unslothai/docs/blob/main/models/qwen3.5)                               | 35B-A3B                                     | [link](https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF)                  | —                                                                               |
|                                                                                                         | 27B                                         | [link](https://huggingface.co/unsloth/Qwen3.5-27B-GGUF)                      | —                                                                               |
|                                                                                                         | 122B-A10B                                   | [link](https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF)                | —                                                                               |
|                                                                                                         | 0.8B                                        | [link](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)                     | —                                                                               |
|                                                                                                         | 2B                                          | [link](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF)                       | —                                                                               |
|                                                                                                         | 4B                                          | [link](https://huggingface.co/unsloth/Qwen3.5-4B-GGUF)                       | —                                                                               |
|                                                                                                         | 9B                                          | [link](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)                       | —                                                                               |
|                                                                                                         | 397B-A17B                                   | [link](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF)                | —                                                                               |
| **Qwen3**                                                                                               | [Coder-Next](/docs/models/qwen3-coder-next) | [link](https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF)                 | —                                                                               |
| [**Qwen-Image**](/docs/models/tutorials/qwen-image-2512)                                                | 2512                                        | [link](https://huggingface.co/unsloth/Qwen-Image-2512-GGUF)                  | —                                                                               |
|                                                                                                         | Edit-2511                                   | [link](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)             | —                                                                               |
| [**Qwen3-VL**](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune) | 2B-Instruct                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-GGUF)             | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-unsloth-bnb-4bit)    |
|                                                                                                         | 2B-Thinking                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-GGUF)             | [link](https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-unsloth-bnb-4bit)    |
|                                                                                                         | 4B-Instruct                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF)             | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-unsloth-bnb-4bit)    |
|                                                                                                         | 4B-Thinking                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-GGUF)             | [link](https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-unsloth-bnb-4bit)    |
|                                                                                                         | 8B-Instruct                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF)             | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit)    |
|                                                                                                         | 8B-Thinking                                 | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-GGUF)             | [link](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-unsloth-bnb-4bit)    |
| **Qwen3-Coder**                                                                                         | 30B-A3B                                     | [link](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF)     | —                                                                               |
|                                                                                                         | 480B-A35B                                   | [link](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF)   | —                                                                               |
| [**Qwen3-2507**](/docs/models/tutorials/qwen3-next)                                                     | 30B-A3B-Instruct                            | [link](https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF)      | —                                                                               |
|                                                                                                         | 30B-A3B-Thinking                            | [link](https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF)      | —                                                                               |
|                                                                                                         | 235B-A22B-Thinking                          | [link](https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF/)   | —                                                                               |
|                                                                                                         | 235B-A22B-Instruct                          | [link](https://huggingface.co/unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF/)   | —                                                                               |
| **Qwen 3**                                                                                              | 0.6 B                                       | [link](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF)                       | [link](https://huggingface.co/unsloth/Qwen3-0.6B-unsloth-bnb-4bit)              |
|                                                                                                         | 1.7 B                                       | [link](https://huggingface.co/unsloth/Qwen3-1.7B-GGUF)                       | [link](https://huggingface.co/unsloth/Qwen3-1.7B-unsloth-bnb-4bit)              |
|                                                                                                         | 4 B                                         | [link](https://huggingface.co/unsloth/Qwen3-4B-GGUF)                         | [link](https://huggingface.co/unsloth/Qwen3-4B-unsloth-bnb-4bit)                |
|                                                                                                         | 8 B                                         | [link](https://huggingface.co/unsloth/Qwen3-8B-GGUF)                         | [link](https://huggingface.co/unsloth/Qwen3-8B-unsloth-bnb-4bit)                |
|                                                                                                         | 14 B                                        | [link](https://huggingface.co/unsloth/Qwen3-14B-GGUF)                        | [link](https://huggingface.co/unsloth/Qwen3-14B-unsloth-bnb-4bit)               |
|                                                                                                         | 30 B-A3B                                    | [link](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF)                    | [link](https://huggingface.co/unsloth/Qwen3-30B-A3B-bnb-4bit)                   |
|                                                                                                         | 32 B                                        | [link](https://huggingface.co/unsloth/Qwen3-32B-GGUF)                        | [link](https://huggingface.co/unsloth/Qwen3-32B-unsloth-bnb-4bit)               |
|                                                                                                         | 235 B-A22B                                  | [link](https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF)                  | —                                                                               |
| **Qwen 2.5 Omni**                                                                                       | 3 B                                         | [link](https://huggingface.co/unsloth/Qwen2.5-Omni-3B-GGUF)                  | —                                                                               |
|                                                                                                         | 7 B                                         | [link](https://huggingface.co/unsloth/Qwen2.5-Omni-7B-GGUF)                  | —                                                                               |
| **Qwen 2.5 VL**                                                                                         | 3 B                                         | [link](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct-GGUF)           | [link](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct-unsloth-bnb-4bit)  |
|                                                                                                         | 7 B                                         | [link](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF)           | [link](https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-unsloth-bnb-4bit)  |
|                                                                                                         | 32 B                                        | [link](https://huggingface.co/unsloth/Qwen2.5-VL-32B-Instruct-GGUF)          | [link](https://huggingface.co/unsloth/Qwen2.5-VL-32B-Instruct-unsloth-bnb-4bit) |
|                                                                                                         | 72 B                                        | [link](https://huggingface.co/unsloth/Qwen2.5-VL-72B-Instruct-GGUF)          | [link](https://huggingface.co/unsloth/Qwen2.5-VL-72B-Instruct-unsloth-bnb-4bit) |
| **Qwen 2.5**                                                                                            | 0.5 B                                       | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit)           |
|                                                                                                         | 1.5 B                                       | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit)           |
|                                                                                                         | 3 B                                         | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct-bnb-4bit)             |
|                                                                                                         | 7 B                                         | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2.5-7B-Instruct-bnb-4bit)             |
|                                                                                                         | 14 B                                        | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2.5-14B-Instruct-bnb-4bit)            |
|                                                                                                         | 32 B                                        | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2.5-32B-Instruct-bnb-4bit)            |
|                                                                                                         | 72 B                                        | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2.5-72B-Instruct-bnb-4bit)            |
| **Qwen 2.5 Coder (128 K)**                                                                              | 0.5 B                                       | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-0.5B-Instruct-128K-GGUF) | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-0.5B-Instruct-bnb-4bit)     |
|                                                                                                         | 1.5 B                                       | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-1.5B-Instruct-128K-GGUF) | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit)     |
|                                                                                                         | 3 B                                         | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-3B-Instruct-128K-GGUF)   | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit)       |
|                                                                                                         | 7 B                                         | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-128K-GGUF)   | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit)       |
|                                                                                                         | 14 B                                        | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-14B-Instruct-128K-GGUF)  | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-14B-Instruct-bnb-4bit)      |
|                                                                                                         | 32 B                                        | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-32B-Instruct-128K-GGUF)  | [link](https://huggingface.co/unsloth/Qwen2.5-Coder-32B-Instruct-bnb-4bit)      |
| **QwQ**                                                                                                 | 32 B                                        | [link](https://huggingface.co/unsloth/QwQ-32B-GGUF)                          | [link](https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit)                 |
| **QVQ (preview)**                                                                                       | 72 B                                        | —                                                                            | [link](https://huggingface.co/unsloth/QVQ-72B-Preview-bnb-4bit)                 |
| **Qwen 2 (chat)**                                                                                       | 1.5 B                                       | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2-1.5B-Instruct-bnb-4bit)             |
|                                                                                                         | 7 B                                         | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2-7B-Instruct-bnb-4bit)               |
|                                                                                                         | 72 B                                        | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2-72B-Instruct-bnb-4bit)              |
| **Qwen 2 VL**                                                                                           | 2 B                                         | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2-VL-2B-Instruct-unsloth-bnb-4bit)    |
|                                                                                                         | 7 B                                         | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2-VL-7B-Instruct-unsloth-bnb-4bit)    |
|                                                                                                         | 72 B                                        | —                                                                            | [link](https://huggingface.co/unsloth/Qwen2-VL-72B-Instruct-bnb-4bit)           |

#### **GLM models:**

| Model     | Variant                                           | GGUF                                                       | Instruct (4-bit) |
| --------- | ------------------------------------------------- | ---------------------------------------------------------- | ---------------- |
| **GLM-5** | [5.3-Flash](/docs/models/glm-5.3-flash)           | [link](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF)  |                  |
|           | [5.3](/docs/models/glm-5.3)                       | [link](https://huggingface.co/unsloth/GLM-5.3-GGUF)        |                  |
| **GLM**   | [4.7-Flash](/docs/models/tutorials/glm-4.7-flash) | [link](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF)  | —                |
|           | [5](/docs/models/tutorials/glm-5)                 | [link](https://huggingface.co/unsloth/GLM-5-GGUF)          | —                |
|           | 4.6V-Flash                                        | [link](https://huggingface.co/unsloth/GLM-4.6V-Flash-GGUF) | —                |
|           | 4.6                                               | [link](https://huggingface.co/unsloth/GLM-4.6-GGUF)        | —                |
|           | 4.5-Air                                           | [link](https://huggingface.co/unsloth/GLM-4.5-Air-GGUF)    | —                |

#### **Mistral models:**

| Model             | Variant           | GGUF                                                                            | Instruct (4-bit)                                                                            |
| ----------------- | ----------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| **Magistral**     | Small (2506)      | [link](https://huggingface.co/unsloth/Magistral-Small-2506-GGUF)                | [link](https://huggingface.co/unsloth/Magistral-Small-2506-unsloth-bnb-4bit)                |
|                   | Small (2509)      | [link](https://huggingface.co/unsloth/Magistral-Small-2509-GGUF)                | [link](https://huggingface.co/unsloth/Magistral-Small-2509-unsloth-bnb-4bit)                |
|                   | Small (2507)      | [link](https://huggingface.co/unsloth/Magistral-Small-2507-GGUF)                | [link](https://huggingface.co/unsloth/Magistral-Small-2507-unsloth-bnb-4bit)                |
| **Mistral Small** | 3.2-24 B (2506)   | [link](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF) | [link](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-unsloth-bnb-4bit) |
|                   | 3.1-24 B (2503)   | [link](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF) | [link](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-unsloth-bnb-4bit) |
|                   | 3-24 B (2501)     | [link](https://huggingface.co/unsloth/Mistral-Small-24B-Instruct-2501-GGUF)     | [link](https://huggingface.co/unsloth/Mistral-Small-24B-Instruct-2501-unsloth-bnb-4bit)     |
|                   | 2409-22 B         | —                                                                               | [link](https://huggingface.co/unsloth/Mistral-Small-Instruct-2409-bnb-4bit)                 |
| **Devstral**      | Small-24 B (2507) | [link](https://huggingface.co/unsloth/Devstral-Small-2507-GGUF)                 | [link](https://huggingface.co/unsloth/Devstral-Small-2507-unsloth-bnb-4bit)                 |
|                   | Small-24 B (2505) | [link](https://huggingface.co/unsloth/Devstral-Small-2505-GGUF)                 | [link](https://huggingface.co/unsloth/Devstral-Small-2505-unsloth-bnb-4bit)                 |
| **Pixtral**       | 12 B (2409)       | —                                                                               | [link](https://huggingface.co/unsloth/Pixtral-12B-2409-bnb-4bit)                            |
| **Mistral NeMo**  | 12 B (2407)       | [link](https://huggingface.co/unsloth/Mistral-Nemo-Instruct-2407-GGUF)          | [link](https://huggingface.co/unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit)                  |
| **Mistral Large** | 2407              | —                                                                               | [link](https://huggingface.co/unsloth/Mistral-Large-Instruct-2407-bnb-4bit)                 |
| **Mistral 7 B**   | v0.3              | —                                                                               | [link](https://huggingface.co/unsloth/mistral-7b-instruct-v0.3-bnb-4bit)                    |
|                   | v0.2              | —                                                                               | [link](https://huggingface.co/unsloth/mistral-7b-instruct-v0.2-bnb-4bit)                    |
| **Mixtral**       | 8 × 7 B           | —                                                                               | [link](https://huggingface.co/unsloth/Mixtral-8x7B-Instruct-v0.1-unsloth-bnb-4bit)          |

#### **Phi models:**

| Model       | Variant          | GGUF                                                             | Instruct (4-bit)                                                             |
| ----------- | ---------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **Phi-4**   | Reasoning-plus   | [link](https://huggingface.co/unsloth/Phi-4-reasoning-plus-GGUF) | [link](https://huggingface.co/unsloth/Phi-4-reasoning-plus-unsloth-bnb-4bit) |
|             | Reasoning        | [link](https://huggingface.co/unsloth/Phi-4-reasoning-GGUF)      | [link](https://huggingface.co/unsloth/phi-4-reasoning-unsloth-bnb-4bit)      |
|             | Mini-Reasoning   | [link](https://huggingface.co/unsloth/Phi-4-mini-reasoning-GGUF) | [link](https://huggingface.co/unsloth/Phi-4-mini-reasoning-unsloth-bnb-4bit) |
|             | Phi-4 (instruct) | [link](https://huggingface.co/unsloth/phi-4-GGUF)                | [link](https://huggingface.co/unsloth/phi-4-unsloth-bnb-4bit)                |
|             | mini (instruct)  | [link](https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF)  | [link](https://huggingface.co/unsloth/Phi-4-mini-instruct-unsloth-bnb-4bit)  |
| **Phi-3.5** | mini             | —                                                                | [link](https://huggingface.co/unsloth/Phi-3.5-mini-instruct-bnb-4bit)        |
| **Phi-3**   | mini             | —                                                                | [link](https://huggingface.co/unsloth/Phi-3-mini-4k-instruct-bnb-4bit)       |
|             | medium           | —                                                                | [link](https://huggingface.co/unsloth/Phi-3-medium-4k-instruct-bnb-4bit)     |

#### **Other (GLM, Orpheus, Smol, Llava etc.) models:**

<table><thead><tr><th>Model</th><th>Variant</th><th width="167">GGUF</th><th>Instruct (4-bit)</th></tr></thead><tbody><tr><td>GLM</td><td>4.5-Air</td><td><a href="https://huggingface.co/unsloth/GLM-4.5-Air-GGUF">link</a></td><td>—</td></tr><tr><td></td><td>4.5</td><td><a href="https://huggingface.co/unsloth/GLM-4.5-GGUF">4.5</a></td><td>—</td></tr><tr><td></td><td>4-32B-0414</td><td><a href="https://huggingface.co/unsloth/GLM-4-32B-0414-GGUF">4-32B-0414</a></td><td>—</td></tr><tr><td><strong>Grok 2</strong></td><td>270B</td><td><a href="https://huggingface.co/unsloth/grok-2-GGUF">link</a></td><td>—</td></tr><tr><td><strong>Baidu-ERNIE</strong></td><td>4.5-21B-A3B-Thinking</td><td><a href="https://huggingface.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF">link</a></td><td>—</td></tr><tr><td>Hunyuan</td><td>A13B</td><td><a href="https://huggingface.co/unsloth/Hunyuan-A13B-Instruct-GGUF">link</a></td><td>—</td></tr><tr><td>Orpheus</td><td>0.1-ft (3B)</td><td><a href="https://huggingface.co/unsloth/orpheus-3b-0.1-ft-GGUF">link</a></td><td><a href="https://huggingface.co/unsloth/orpheus-3b-0.1-ft-unsloth-bnb-4bit">link</a></td></tr><tr><td><strong>LLava</strong></td><td>1.5 (7 B)</td><td>—</td><td><a href="https://huggingface.co/unsloth/llava-1.5-7b-hf-bnb-4bit">link</a></td></tr><tr><td></td><td>1.6 Mistral (7 B)</td><td>—</td><td><a href="https://huggingface.co/unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit">link</a></td></tr><tr><td><strong>TinyLlama</strong></td><td>Chat</td><td>—</td><td><a href="https://huggingface.co/unsloth/tinyllama-chat-bnb-4bit">link</a></td></tr><tr><td><strong>SmolLM 2</strong></td><td>135 M</td><td><a href="https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF">link</a></td><td><a href="https://huggingface.co/unsloth/SmolLM2-135M-Instruct-bnb-4bit">link</a></td></tr><tr><td></td><td>360 M</td><td><a href="https://huggingface.co/unsloth/SmolLM2-360M-Instruct-GGUF">link</a></td><td><a href="https://huggingface.co/unsloth/SmolLM2-360M-Instruct-bnb-4bit">link</a></td></tr><tr><td></td><td>1.7 B</td><td><a href="https://huggingface.co/unsloth/SmolLM2-1.7B-Instruct-GGUF">link</a></td><td><a href="https://huggingface.co/unsloth/SmolLM2-1.7B-Instruct-bnb-4bit">link</a></td></tr><tr><td><strong>Zephyr-SFT</strong></td><td>7 B</td><td>—</td><td><a href="https://huggingface.co/unsloth/zephyr-sft-bnb-4bit">link</a></td></tr><tr><td><strong>Yi</strong></td><td>6 B (v1.5)</td><td>—</td><td><a href="https://huggingface.co/unsloth/Yi-1.5-6B-bnb-4bit">link</a></td></tr><tr><td></td><td>6 B (v1.0)</td><td>—</td><td><a href="https://huggingface.co/unsloth/yi-6b-bnb-4bit">link</a></td></tr><tr><td></td><td>34 B (chat)</td><td>—</td><td><a href="https://huggingface.co/unsloth/yi-34b-chat-bnb-4bit">link</a></td></tr><tr><td></td><td>34 B (base)</td><td>—</td><td><a href="https://huggingface.co/unsloth/yi-34b-bnb-4bit">link</a></td></tr></tbody></table>


# Fine-tuning for Beginners

If you're a beginner, here might be the first questions you'll ask before your first fine-tune. You can also always ask our community by joining our [Reddit page](https://www.reddit.com/r/unsloth/).

<table data-view="cards"><thead><tr><th data-type="content-ref"></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><a href="/docs/models/tutorials">Complete LLM Directory</a></td><td>See every model you can run / train with Unsloth.</td><td>How to run GGUFs or train LLMs?</td><td><a href="/docs/models/tutorials">Complete LLM Directory</a></td></tr><tr><td><a href="/docs/get-started/fine-tuning-llms-guide">Fine-tuning Guide</a></td><td>Step-by-step on how to fine-tune!</td><td>Learn the core basics of training.</td><td><a href="/docs/get-started/fine-tuning-llms-guide">Fine-tuning Guide</a></td></tr><tr><td><a href="/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use">What Model Should I Use?</a></td><td>Instruct or Base Model?</td><td>How big should my dataset be?</td><td><a href="/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use">What Model Should I Use?</a></td></tr><tr><td><a href="/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me">FAQ + Is Fine-tuning Right For Me?</a></td><td>What can fine-tuning do for me?</td><td>RAG vs. Fine-tuning?</td><td><a href="/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me">FAQ + Is Fine-tuning Right For Me?</a></td></tr><tr><td><a href="/docs/get-started/install">Installation</a></td><td>How do I install Unsloth locally?</td><td>How to update Unsloth?</td><td><a href="/docs/get-started/install">Installation</a></td></tr><tr><td><a href="/docs/get-started/fine-tuning-llms-guide/datasets-guide">Datasets Guide</a></td><td>How do I structure/prepare my dataset?</td><td>How do I collect data?</td><td></td></tr><tr><td><a href="/docs/get-started/fine-tuning-for-beginners/unsloth-requirements">Unsloth Requirements</a></td><td>Does Unsloth work on my GPU?</td><td>How much VRAM will I need?</td><td><a href="/docs/get-started/fine-tuning-for-beginners/unsloth-requirements">Unsloth Requirements</a></td></tr><tr><td><a href="/docs/basics/inference-and-deployment">Inference &amp; Deployment</a></td><td>How do I save my model locally?</td><td>How do I run my model via Ollama or vLLM?</td><td><a href="/docs/basics/inference-and-deployment">Inference &amp; Deployment</a></td></tr><tr><td><a href="/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide">Hyperparameters Guide</a></td><td>What happens when I change a parameter?</td><td>What parameters should I change?</td><td></td></tr></tbody></table>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-559e7890f607e34fd6004517296e65e942c93b41%2FLarge%20sloth%20Question%20mark.png?alt=media" alt="" width="188"><figcaption></figcaption></figure>


# Unsloth Requirements

Here are Unsloth's requirements including system and GPU VRAM requirements.

Unsloth can be used in three ways: our [Unsloth Desktop](/docs/desktop) app, our [Unsloth Studio](/docs/new/studio/install) web UI, or through [Unsloth Core](#unsloth-core-requirements), the original code-based version. Each has different requirements.

## **Unsloth Requirements**

* **Mac:** Training, MLX and GGUF inference are ALL supported.
* **CPU: Unsloth still works without a GPU**, for Chat + Data Recipes.
* **Training:** Works on **NVIDIA**, **AMD**, **Intel** GPUs and **Mac** devices

### <i class="fa-windows">:windows:</i> Window**s**

Unsloth Studio works directly on Windows without WSL. To train models, make sure your system satisfies these requirements:

**Requirements**

* Windows 10 or Windows 11 (64-bit)
* NVIDIA GPU with drivers installed
* **App Installer** (includes `winget`): [here](https://learn.microsoft.com/en-us/windows/msix/app-installer/install-update-app-installer)
* **Git**: `winget install --id Git.Git -e --source winget`
* **Python**: version 3.11 up to, but not including, 3.14
* Work inside a Python environment such as **uv**, **venv**, or **conda/mamba**

### <i class="fa-apple">:apple:</i> MacOS

Unsloth Studio works on MacOS device with full support for training, MLX and GGUF inference.

* macOS 12 Monterey or newer (Intel or Apple Silicon)
* Install Homebrew: [here](https://brew.sh/)
* Git: `brew install git`&#x20;
* cmake: `brew install cmake`&#x20;
* openssl: `brew install openssl`
* Python: version 3.11 up to, but not including, 3.14
* Work inside a Python environment such as **uv**, **venv**, or **conda/mamba**

### <i class="fa-linux">:linux:</i> Linux & WSL

* Ubuntu 20.04+ or similar distro (64-bit)
* NVIDIA GPU with drivers installed
* CUDA toolkit (12.4+ recommended, 12.8+ for blackwell)
* Git: `sudo apt install git`
* Python: version 3.11 up to, but not including, 3.14
* Work inside a Python environment such as **uv**, **venv**, or **conda/mamba**

### <i class="fa-microchip">:microchip:</i> CPU only

Unsloth Studio supports CPU devices for [Chat](#run-models-locally) for GGUF models and [Data Recipes](/docs/new/studio/data-recipe) ([Export](/docs/new/studio/export) coming very soon)

* Same as the ones mentioned above for Linux (except for NVIDIA GPU drivers) and MacOS.

### **Training**

Unsloth Studio Training currently works on NVIDIA, [AMD](/docs/get-started/install/amd), MLX, Intel devices. You can still use the [original Unsloth Core](#unsloth-requirements) to train on AMD and Intel devices. **Python 3.11–3.13** is required.

| Requirement      | Linux / WSL                              | Windows                                       |
| ---------------- | ---------------------------------------- | --------------------------------------------- |
| **Git**          | Usually preinstalled                     | Installed by setup script (`winget`)          |
| **CMake**        | Preinstalled or `sudo apt install cmake` | Installed by setup script (`winget`)          |
| **C++ compiler** | `build-essential`                        | Visual Studio Build Tools 2022                |
| **CUDA Toolkit** | Optional; `nvcc` auto-detected           | Installed by setup script (matched to driver) |

## Unsloth Core Requirements

* **Operating System**: Works on Linux and [Windows](https://docs.unsloth.ai/get-started/install-and-update/windows-installation)
* Supports NVIDIA GPUs since 2018+ including [Blackwell RTX 50](/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) and [DGX Spark](/docs/blog/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth)
* Minimum CUDA Capability 7.0 (V100, T4, Titan V, RTX 20 & 50, A100, H100, L40 etc) [Check your GPU!](https://developer.nvidia.com/cuda-gpus) GTX 1070, 1080 works, but is slow.
* The official [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker Hub
  * [Docker](/docs/get-started/install/docker)
* Unsloth works on [AMD](/docs/get-started/install/amd) and [Intel](/docs/get-started/install/intel) GPUs (follow our [specific guides](/docs/get-started/install)). Apple/Silicon/MLX is in the works
* Your device should have `xformers`, `torch`, `BitsandBytes` and `triton` support.
* If you have different versions of torch, transformers etc., `pip install unsloth` will automatically install all the latest versions of those libraries so you don't need to worry about version compatibility.

{% hint style="info" %}
Python 3.13 is supported!
{% endhint %}

### Fine-tuning VRAM requirements:

How much GPU memory do I need for LLM fine-tuning using Unsloth?

{% hint style="info" %}
A common issue when you OOM or run out of memory is because you set your batch size too high. Set it to 1, 2, or 3 to use less VRAM.

**For context length benchmarks, see** [**here**](/docs/basics/unsloth-benchmarks#context-length-benchmarks)**.**
{% endhint %}

Check this table for VRAM requirements sorted by model parameters and fine-tuning method. QLoRA uses 4-bit, LoRA uses 16-bit. Keep in mind that sometimes more VRAM is required depending on the model so these numbers are the absolute minimum:

| Model parameters | QLoRA (4-bit) VRAM | LoRA (16-bit) VRAM |
| ---------------- | ------------------ | ------------------ |
| 3B               | 3.5 GB             | 8 GB               |
| 7B               | 5 GB               | 19 GB              |
| 8B               | 6 GB               | 22 GB              |
| 9B               | 6.5 GB             | 24 GB              |
| 11B              | 7.5 GB             | 29 GB              |
| 14B              | 8.5 GB             | 33 GB              |
| 27B              | 22GB               | 64GB               |
| 32B              | 26 GB              | 76 GB              |
| 40B              | 30GB               | 96GB               |
| 70B              | 41 GB              | 164 GB             |
| 81B              | 48GB               | 192GB              |
| 90B              | 53GB               | 212GB              |
| 405B             | 237 GB             | 950 GB             |


# FAQ + Is Fine-tuning Right For Me?

If you're stuck on if fine-tuning is right for you, see here! Learn about fine-tuning misconceptions, how it compared to RAG and more:

## Understanding Fine-Tuning

Fine-tuning an LLM customizes its behavior, deepens its domain expertise, and optimizes its performance for specific tasks. By refining a pre-trained model (e.g. *Llama-3.1-8B*) with specialized data, you can:

* **Update Knowledge** – Introduce new, domain-specific information that the base model didn’t originally include.
* **Customize Behavior** – Adjust the model’s tone, personality, or response style to fit specific needs or a brand voice.
* **Optimize for Tasks** – Improve accuracy and relevance on particular tasks or queries your use-case requires.

Think of fine-tuning as creating a specialized expert out of a generalist model. Some debate whether to use Retrieval-Augmented Generation (RAG) instead of fine-tuning, but fine-tuning can incorporate knowledge and behaviors directly into the model in ways RAG cannot. In practice, combining both approaches yields the best results - leading to greater accuracy, better usability, and fewer hallucinations.

### Real-World Applications of Fine-Tuning

Fine-tuning can be applied across various domains and needs. Here are a few practical examples of how it makes a difference:

* **Sentiment Analysis for Finance** – Train an LLM to determine if a news headline impacts a company positively or negatively, tailoring its understanding to financial context.
* **Customer Support Chatbots** – Fine-tune on past customer interactions to provide more accurate and personalized responses in a company’s style and terminology.
* **Legal Document Assistance** – Fine-tune on legal texts (contracts, case law, regulations) for tasks like contract analysis, case law research, or compliance support, ensuring the model uses precise legal language.

## The Benefits of Fine-Tuning

Fine-tuning offers several notable benefits beyond what a base model or a purely retrieval-based system can provide:

#### Fine-Tuning vs. RAG: What’s the Difference?

Fine-tuning can do mostly everything RAG can - but not the other way around. During training, fine-tuning embeds external knowledge directly into the model. This allows the model to handle niche queries, summarize documents, and maintain context without relying on an outside retrieval system. That’s not to say RAG lacks advantages as it is excels at accessing up-to-date information from external databases. It is in fact possible to retrieve fresh data with fine-tuning as well, however it is better to combine RAG with fine-tuning for efficiency.

#### Task-Specific Mastery

Fine-tuning deeply integrates domain knowledge into the model. This makes it highly effective at handling structured, repetitive, or nuanced queries, scenarios where RAG-alone systems often struggle. In other words, a fine-tuned model becomes a specialist in the tasks or content it was trained on.

#### Independence from Retrieval

A fine-tuned model has no dependency on external data sources at inference time. It remains reliable even if a connected retrieval system fails or is incomplete, because all needed information is already within the model’s own parameters. This self-sufficiency means fewer points of failure in production.

#### Faster Responses

Fine-tuned models don’t need to call out to an external knowledge base during generation. Skipping the retrieval step means they can produce answers much more quickly. This speed makes fine-tuned models ideal for time-sensitive applications where every second counts.

#### Custom Behavior and Tone

Fine-tuning allows precise control over how the model communicates. This ensures the model’s responses stay consistent with a brand’s voice, adhere to regulatory requirements, or match specific tone preferences. You get a model that not only knows *what* to say, but *how* to say it in the desired style.

#### Reliable Performance

Even in a hybrid setup that uses both fine-tuning and RAG, the fine-tuned model provides a reliable fallback. If the retrieval component fails to find the right information or returns incorrect data, the model’s built-in knowledge can still generate a useful answer. This guarantees more consistent and robust performance for your system.

## Common Misconceptions

Despite fine-tuning’s advantages, a few myths persist. Let’s address two of the most common misconceptions about fine-tuning:

### Does Fine-Tuning Add New Knowledge to a Model?

**Yes - it absolutely can.** A common myth suggests that fine-tuning doesn’t introduce new knowledge, but in reality it does. If your fine-tuning dataset contains new domain-specific information, the model will learn that content during training and incorporate it into its responses. In effect, fine-tuning *can and does* teach the model new facts and patterns from scratch.

### Is RAG Always Better Than Fine-Tuning?

**Not necessarily.** Many assume RAG will consistently outperform a fine-tuned model, but that’s not the case when fine-tuning is done properly. In fact, a well-tuned model often matches or even surpasses RAG-based systems on specialized tasks. Claims that “RAG is always better” usually stem from fine-tuning attempts that weren’t optimally configured - for example, using incorrect [LoRA parameters](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) or insufficient training.

Unsloth takes care of these complexities by automatically selecting the best parameter configurations for you. All you need is a good-quality dataset, and you'll get a fine-tuned model that performs to its fullest potential.

### Is Fine-Tuning Expensive?

**Not at all!** While full fine-tuning or pretraining can be costly, these are not necessary (pretraining is especially not necessary). In most cases, LoRA or QLoRA fine-tuning can be done for minimal cost. In fact, with Unsloth’s [free notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks) for Colab or Kaggle, you can fine-tune models without spending a dime. Better yet, you can even fine-tune locally on your own device.

## FAQ:

### Why You Should Combine RAG & Fine-Tuning

Instead of choosing between RAG and fine-tuning, consider using **both** together for the best results. Combining a retrieval system with a fine-tuned model brings out the strengths of each approach. Here’s why:

* **Task-Specific Expertise** – Fine-tuning excels at specialized tasks or formats (making the model an expert in a specific area), while RAG keeps the model up-to-date with the latest external knowledge.
* **Better Adaptability** – A fine-tuned model can still give useful answers even if the retrieval component fails or returns incomplete information. Meanwhile, RAG ensures the system stays current without requiring you to retrain the model for every new piece of data.
* **Efficiency** – Fine-tuning provides a strong foundational knowledge base within the model, and RAG handles dynamic or quickly-changing details without the need for exhaustive re-training from scratch. This balance yields an efficient workflow and reduces overall compute costs.

### LoRA vs. QLoRA: Which One to Use?

When it comes to implementing fine-tuning, two popular techniques can dramatically cut down the compute and memory requirements: **LoRA** and **QLoRA**. Here’s a quick comparison of each:

* **LoRA (Low-Rank Adaptation)** – Fine-tunes only a small set of additional “adapter” weight matrices (in 16-bit precision), while leaving most of the original model unchanged. This significantly reduces the number of parameters that need updating during training.
* **QLoRA (Quantized LoRA)** – Combines LoRA with 4-bit quantization of the model weights, enabling efficient fine-tuning of very large models on minimal hardware. By using 4-bit precision where possible, it dramatically lowers memory usage and compute overhead.

We recommend starting with **QLoRA**, as it’s one of the most efficient and accessible methods available. Thanks to Unsloth’s [dynamic 4-bit](https://unsloth.ai/blog/dynamic-4bit) quants, the accuracy loss compared to standard 16-bit LoRA fine-tuning is now negligible.

### Experimentation is Key

There’s no single “best” approach to fine-tuning - only best practices for different scenarios. It’s important to experiment with different methods and configurations to find what works best for your dataset and use case. A great starting point is **QLoRA (4-bit)**, which offers a very cost-effective, resource-friendly way to fine-tune models without heavy computational requirements.

{% content-ref url="/pages/y6obKRSk8TwyjIrCjuGE" %}
[Hyperparameters Guide](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide)
{% endcontent-ref %}


# Unsloth Notebooks

Fine-tuning notebooks: Explore the Unsloth catalog.

Train your own model with our notebooks, powered by free GPU compute. Click Run all (or save locally), add your dataset, train, and deploy. You can use any model in the notebooks.

<a href="/docs/get-started/unsloth-notebooks#grpo-reasoning-rl" class="button secondary">GRPO (RL)</a><a href="/docs/get-started/unsloth-notebooks#text-to-speech-tts" class="button secondary">Text-to-speech</a><a href="/docs/get-started/unsloth-notebooks#vision-multimodal" class="button secondary">Vision</a><a href="/docs/get-started/unsloth-notebooks#embedding-models" class="button secondary">Embedding</a><a href="/docs/get-started/unsloth-notebooks#kaggle-notebooks" class="button secondary">Kaggle</a>

Also see our GitHub repo for our notebooks: [github.com/unslothai/notebooks](https://github.com/unslothai/notebooks/)

## Colab notebooks

**Introducing our** [**Unsloth Studio**](/docs/new/studio)✨ **notebook.** Train and run models under 22B parameters:

{% embed url="<https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb>" %}

### Standard SFT notebooks:

* [**Qwen3.8 27B**](/docs/models/qwen3.8/train): [Conversational (Kaggle)](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen3.8_\(27B\)-Conversational.ipynb\&accelerator=nvidiaTeslaT4)
* [**Qwen3.5 4B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)-Decision.ipynb) - Decision model training
* [**Muse Glimmer (30B)**](/docs/models/muse-glimmer/train) **•** [Conversational (Kaggle)](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Muse_Glimmer_\(30B\)-Conversational.ipynb\&accelerator=nvidiaTeslaT4)  • [Vision (Kaggle)](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Muse_Glimmer_\(30B\)-Vision.ipynb\&accelerator=nvidiaTeslaT4)
* [**Gemma 4**](/docs/models/gemma-4/train)**:** [E4B **(Vision)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E4B\)-Vision.ipynb) **•** [E2B **(Text)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)-Text.ipynb) **•** [E2B **(Audio)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)-Audio.ipynb) **•** [**31B** (Kaggle)](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) **•** [**Inference**](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb)
* [**Qwen3.5**](/docs/models/qwen3.5/fine-tune)**:** [**0.8B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(0_8B\)_Vision.ipynb) • [**2B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(2B\)_Vision.ipynb) • [**4B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)_Vision.ipynb)
* [gpt-oss (20b)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-Fine-tuning.ipynb) • [Inference](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GPT_OSS_MXFP4_\(20B\)-Inference.ipynb) • [Fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-Fine-tuning.ipynb)
* [EmbeddingGemma (300M)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_\(300M\).ipynb) • [EmbeddingGemma-2 Image Embeddings](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_\(300M\)-Image_Text.ipynb)
* [Qwen3 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(14B\)-Reasoning-Conversational.ipynb) • [**Qwen3-VL (8B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision.ipynb)
* [**Qwen3-2507-4B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-Instruct.ipynb) • [Thinking](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-Thinking.ipynb) • [Instruct](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-Instruct.ipynb)
* [Gemma 3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\).ipynb) • [Text](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\).ipynb) • [Vision](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision.ipynb) • [270M](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(270M\).ipynb) • [**FunctionGemma**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\).ipynb)
* [Gemma 3n (E4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Conversational.ipynb) • [Text](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Conversational.ipynb) • [Vision](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Vision.ipynb) • [Audio](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Audio.ipynb)
* [**Mistral Ministral 3**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_\(3B\)_Vision.ipynb)
* [**DeepSeek-OCR 2**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_2_\(3B\).ipynb)&#x20;
* [IBM Granite-4.0-H](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb)
* [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb)
* [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_\(8B\)-Alpaca.ipynb) • [Llama 3.2 (1B + 3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(1B_and_3B\)-Conversational.ipynb)

### Decision models

* [**Qwen3.5 4B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)-Decision.ipynb) - new - Typed Decisions

### GRPO (Reasoning RL):

* [**Muse Glimmer (30B)**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Muse_Glimmer_\(30B\)-GRPO.ipynb\&accelerator=nvidiaTeslaT4)
* [**Gemma 4 E2B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)_Reinforcement_Learning_Sudoku_Game.ipynb) - new
* [**Qwen3.5 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)_Vision_GRPO.ipynb) - Vision GRPO
* [PPO with Llama](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_%281B%29-PPO.ipynb#scrollTo=zsiKGWFhYemJ) - new PPO notebook
* [gpt-oss-20b](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) (automatic kernels creation)
* [Mistral Ministral 3](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_\(3B\)_Reinforcement_Learning_Sudoku_Game.ipynb) (solving sodoku) - new
* [Qwen3-8B - **FP8**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) (L4) - new
* [Llama-3.2-1B - **FP8**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama_FP8_GRPO.ipynb) (L4) - new
* [gpt-oss-20b](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_\(20B\)_Reinforcement_Learning_2048_Game.ipynb) (auto win 2048 game)&#x20;
* [Qwen3-VL (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb) - Vision GSPO
* [Qwen3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-GRPO.ipynb) - Advanced GRPO LoRA
* [Gemma 3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision-GRPO.ipynb) - Vision GSPO
* [gpt-oss-20b](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_\(20B\)_Reinforcement_Learning_2048_Game.ipynb) (2048 OpenEnv example)
* [DeepSeek-R1-0528-Qwen3 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_\(8B\)_GRPO.ipynb) (for multilingual usecase)
* [Gemma 3 (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(1B\)-GRPO.ipynb)
* [Llama 3.2 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_\(3B\)_GRPO_LoRA.ipynb) - Advanced GRPO LoRA
* [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_\(8B\)-GRPO.ipynb)
* [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_\(14B\)-GRPO.ipynb)
* [Mistral v0.3 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-GRPO.ipynb)
* [NeMo Gym Multi Agents Environment ](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Multi-Environment.ipynb)(Multiple agentic environments)

### Text-to-Speech (TTS):

* [Sesame-CSM (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Sesame_CSM_\(1B\)-TTS.ipynb)
* [Orpheus-TTS (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_\(3B\)-TTS.ipynb)
* [Whisper Large V3](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb) - Speech-to-Text (STT)
* [Llasa-TTS (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llasa_TTS_\(1B\).ipynb)
* [Spark-TTS (0.5B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Spark_TTS_\(0_5B\).ipynb)
* [Oute-TTS (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Oute_TTS_\(1B\).ipynb)

**Speech-to-Text (SST):**

* [**Gemma 4 (E2B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)-Audio.ipynb) **- Audio - new**
* [Whisper-Large-V3](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb)
* [Gemma 3n (E4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Audio.ipynb) - Audio

### Vision (Multimodal):

* [**Gemma 4**](/docs/models/gemma-4/train)**:** [E2B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)-Vision.ipynb) **•** [**31B** (Kaggle)](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) - new
* [**Qwen3.5**](/docs/models/qwen3.5/fine-tune)**:** [**0.8B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(0_8B\)_Vision.ipynb) • [**2B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(2B\)_Vision.ipynb) • [**4B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)_Vision.ipynb) - new
* [**Qwen3-VL (8B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision.ipynb)
* [**Mistral Ministral 3**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_\(3B\)_Vision.ipynb)
* [**DeepSeek-OCR**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_\(3B\).ipynb)
* [**Paddle-OCR (1B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Paddle_OCR_\(1B\)_Vision.ipynb)
* [Gemma 3n (E4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Vision.ipynb)
* [Gemma 3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision.ipynb)
* [Llama 3.2 Vision (11B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(11B\)-Vision.ipynb)
* [Qwen2.5-VL (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_VL_\(7B\)-Vision.ipynb)
* [Pixtral (12B) 2409](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Pixtral_\(12B\)-Vision.ipynb)
* [Qwen3-VL](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb) - Vision GSPO - new
* [Qwen2.5-VL](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb) - Vision GSPO
* [Gemma 3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision-GRPO.ipynb) - Vision GSPO

### Embedding models:

* [EmbeddingGemma (300M)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_\(300M\).ipynb) - new
* **EmbeddingGemma2** - new - [Multimodal search](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_\(300M\)-Multimodal_Search.ipynb) • [Image](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_\(300M\)-Image_Text.ipynb) • [Text](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_\(300M\).ipynb) • [Audio](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_\(300M\)-Audio.ipynb)
* [Qwen3-Embedding 4B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_\(4B\).ipynb) - new
* [Qwen3-Embedding 0.6B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_Embedding_\(0_6B\).ipynb) - new
* [BGE M3](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/BGE_M3.ipynb) - new
* [ModernBERT-large](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) - new
* [All-MiniLM-L6-v2](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/All_MiniLM_L6_v2.ipynb) - new
* [GTE ModernBert](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/ModernBert.ipynb) - new

### Large LLMs:

**Notebooks for large models:** These exceed Colab’s free 15 GB VRAM tier. With Colab’s new 80 GB GPUs, you can fine-tune 120B parameter models.

{% hint style="info" %}
Colab subscription or credits are required. We **don't** earn anything from these notebooks.
{% endhint %}

* [**Gemma-4-31B**](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) - new and **FREE**
* [**DiffusionGemma**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DiffusionGemma_\(26B-A4B\)-Sudoku.ipynb) - new
* [Gemma-4-26B-A4B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(26B_A4B\)-Vision.ipynb) - new
* [Gemma-4-31B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(31B\)-Vision.ipynb) - new
* [Qwen3.5-35B-A3B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_MoE.ipynb)
* [Qwen3.5‑27B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen_3_5_27B_A100\(80GB\).ipynb)
* [GLM-4.7-Flash](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GLM_Flash_A100\(80GB\).ipynb)
* [gpt-oss-20b (500K context)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_\(20B\)_500K_Context_Fine_tuning.ipynb)
* [Qwen3-30B-A3B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_MoE.ipynb)
* [Nemotron-3-Nano-30B-A3B LoRA notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Nemotron-3-Nano-30B-A3B_A100.ipynb)&#x20;
* [NeMo Gym Sudoku GRPO notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Sudoku.ipynb)
* [NeMo Gym Multi Environment GRPO Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/NeMo-Gym-Multi-Environment.ipynb)
* [gpt-oss-120b](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(120B\)_A100-Fine-tuning.ipynb)
* [Qwen3 (32B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(32B\)_A100-Reasoning-Conversational.ipynb)
* [Llama 3.3 (70B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.3_\(70B\)_A100-Conversational.ipynb)
* [Gemma 3 (27B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(27B\)_A100-Conversational.ipynb)&#x20;
* [Baidu ERNIE 4.5 VL (28B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/ERNIE_4_5_VL_28B_A3B_PT_Vision.ipynb) - new

### Other important notebooks:

* [**Customer support agent**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb)
* [Mistral Ministral 3](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_\(3B\)_Reinforcement_Learning_Sudoku_Game.ipynb) - new (solving sodoku)
* [Deploy on LM Studio ](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-LMStudio.ipynb)- new
* [Quantization-Aware Training](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)_Instruct-QAT.ipynb) (QAT) - new
* [Phone Deployment ](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(0_6B\)-Phone_Deployment.ipynb)- new
* [Reason before **Tool Calling** notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\).ipynb) - new
* [Mobile Actions notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Mobile-Actions.ipynb) - new
* [**Automatic Kernel Creation**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) with RL
* [**ModernBERT-large**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/bert_classification.ipynb) **- new** Aug 19
* [**Synthetic Data Generation Llama 3.2 (3B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_\(3B\).ipynb)
* [gpt-oss-20b (500K context)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt_oss_\(20B\)_500K_Context_Fine_tuning.ipynb) - new (A100)
* [**Tool Calling**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_\(1.5B\)-Tool_Calling.ipynb)
* [Mistral v0.3 Instruct (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-Conversational.ipynb)
* [Ollama](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Ollama.ipynb)
* [ORPO](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-ORPO.ipynb)
* [Continued Pretraining](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-CPT.ipynb)
* [DPO Zephyr](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_\(7B\)-DPO.ipynb)
* [***Inference only***](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_\(8B\)-Inference.ipynb)
* [Llama 3 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Alpaca.ipynb)

### Specific use-case notebooks:

* [Phone Deployment ](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(0_6B\)-Phone_Deployment.ipynb)- new
* [Deploy on LM Studio ](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-LMStudio.ipynb)- new
* [Reason before **Tool Calling**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\).ipynb) - new
* [Mobile Actions](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Mobile-Actions.ipynb) - new
* [**Customer support agent**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb)
* [Quantization-Aware Training](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)_Instruct-QAT.ipynb) (QAT) - new
* [**Automatic Kernel Creation**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) with RL **- new**
* [DPO Zephyr](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_\(7B\)-DPO.ipynb)
* [BERT - Text Classification](https://colab.research.google.com/github/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb) - (AutoModelForSequenceClassification)
* [Ollama](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Ollama.ipynb)
* [**Tool Calling**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_\(1.5B\)-Tool_Calling.ipynb)
* [Continued Pretraining (CPT)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-CPT.ipynb)
* [Multiple Datasets](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) by Flail
* [KTO](https://colab.research.google.com/drive/1MRgGtLWuZX4ypSfGguFgC-IblTvO2ivM?usp=sharing) by Jeffrey
* [Inference chat UI](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Unsloth_Studio.ipynb)
* [Conversational](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(1B_and_3B\)-Conversational.ipynb)
* [ChatML](https://colab.research.google.com/drive/15F1xyn8497_dUbxZP4zWmPZ3PJx1Oymv?usp=sharing)
* [Text Completion](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_\(7B\)-Text_Completion.ipynb)

### Rest of notebooks:

* [Qwen2.5 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_\(3B\)-GRPO.ipynb)
* [Gemma 2 (9B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_\(9B\)-Alpaca.ipynb)
* [Mistral NeMo (12B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Nemo_\(12B\)-Alpaca.ipynb)
* [Phi-3.5 (mini)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3.5_Mini-Conversational.ipynb)
* [Phi-3 (medium)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3_Medium-Conversational.ipynb)
* [Gemma 2 (2B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_\(2B\)-Alpaca.ipynb)
* [Qwen 2.5 Coder (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_Coder_\(14B\)-Conversational.ipynb)
* [Mistral Small (22B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_Small_\(22B\)-Alpaca.ipynb)
* [TinyLlama](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/TinyLlama_\(1.1B\)-Alpaca.ipynb)
* [CodeGemma (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/CodeGemma_\(7B\)-Conversational.ipynb)
* [Mistral v0.3 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-Alpaca.ipynb)
* [Qwen2 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_\(7B\)-Alpaca.ipynb)

## Kaggle notebooks

#### Standard notebooks:

* [**Gemma-4-31B** (Kaggle)](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) - new and **FREE**
* [**gpt-oss (20B)**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-gpt-oss-\(20B\)-Fine-tuning.ipynb\&accelerator=nvidiaTeslaT4)
* [Gemma 3n (E4B)](https://www.kaggle.com/code/danielhanchen/gemma-3n-4b-multimodal-finetuning-inference)
* [Qwen3 (14B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen3_\(14B\).ipynb)
* [Magistral-2509 (24B)](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_\(24B\)-Reasoning-Conversational.ipynb\&accelerator=nvidiaTeslaT4)
* [Gemma 3 (4B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_\(4B\).ipynb)
* [Phi-4 (14B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4-Conversational.ipynb)
* [Llama 3.1 (8B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_\(8B\)-Alpaca.ipynb)
* [Llama 3.2 (1B + 3B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_\(1B_and_3B\)-Conversational.ipynb)
* [Qwen 2.5 (7B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_\(7B\)-Alpaca.ipynb)

#### GRPO (Reasoning) notebooks:

* [**Qwen2.5-VL**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2_5_7B_VL_GRPO.ipynb\&accelerator=nvidiaTeslaT4) - Vision GRPO - new
* [Qwen3 (4B)](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen3_\(4B\)-GRPO.ipynb\&accelerator=nvidiaTeslaT4)
* [Gemma 3 (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma3_\(1B\)-GRPO.ipynb)
* [Llama 3.1 (8B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_\(8B\)-GRPO.ipynb)
* [Phi-4 (14B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Phi_4_\(14B\)-GRPO.ipynb)
* [Qwen 2.5 (3B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_\(3B\)-GRPO.ipynb)

#### Text-to-Speech (TTS) notebooks:

* [Sesame-CSM (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Sesame_CSM_\(1B\)-TTS.ipynb)
* [Orpheus-TTS (3B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Orpheus_\(3B\)-TTS.ipynb)
* [Whisper Large V3](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Whisper.ipynb) – Speech-to-Text
* [Llasa-TTS (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llasa_TTS_\(1B\).ipynb)
* [Spark-TTS (0.5B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Spark_TTS_\(0_5B\).ipynb)
* [Oute-TTS (1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Oute_TTS_\(1B\).ipynb)

#### Vision (Multimodal) notebooks:

* [Llama 3.2 Vision (11B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.2_\(11B\)-Vision.ipynb)
* [Qwen 2.5-VL (7B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_VL_\(7B\)-Vision.ipynb)
* [Pixtral (12B) 2409](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Pixtral_\(12B\)-Vision.ipynb)

#### Specific use-case notebooks:

* [Tool Calling](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2.5_Coder_\(1.5B\)-Tool_Calling.ipynb\&accelerator=nvidiaTeslaT4)
* [ORPO](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_\(8B\)-ORPO.ipynb)
* [Continued Pretraining](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_v0.3_\(7B\)-CPT.ipynb)
* [DPO Zephyr](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Zephyr_\(7B\)-DPO.ipynb)
* [Inference only](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3.1_\(8B\)-Inference.ipynb)
* [Ollama](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Llama3_\(8B\)-Ollama.ipynb)
* [Text Completion](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_\(7B\)-Text_Completion.ipynb)
* [CodeForces-cot (Reasoning)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeForces-cot-Finetune_for_Reasoning_on_CodeForces.ipynb)
* [Unsloth Studio (chat UI)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Unsloth_Studio.ipynb)

#### Rest of notebooks:

* [Gemma 2 (9B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_\(9B\)-Alpaca.ipynb)
* [Gemma 2 (2B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Gemma2_\(2B\)-Alpaca.ipynb)
* [CodeGemma (7B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-CodeGemma_\(7B\)-Conversational.ipynb)
* [Mistral NeMo (12B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Nemo_\(12B\)-Alpaca.ipynb)
* [Mistral Small (22B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-Mistral_Small_\(22B\)-Alpaca.ipynb)
* [TinyLlama (1.1B)](https://www.kaggle.com/notebooks/welcome?src=https%3A%2F%2Fgithub.com%2Funslothai/notebooks/blob/main/nb/Kaggle-TinyLlama_\(1.1B\)-Alpaca.ipynb)

To view a complete list of all our Kaggle notebooks, [click here](https://github.com/unslothai/notebooks#-kaggle-notebooks).

{% hint style="info" %}
Feel free to contribute to the notebooks by visiting our [repo](https://github.com/unslothai/notebooks)!
{% endhint %}


# Fine-tuning LLMs Guide

Learn all the basics and best practices of fine-tuning. Beginner-friendly.

## 1. What Is Fine-tuning?

Fine-tuning / training / post-training models customizes its behavior, enhances + injects knowledge, and optimizes performance for domains and specific tasks. For example:

* OpenAI’s **GPT-5** was post-trained to improve instruction following and helpful chat behavior.
* The standard method of post training is called Supervised Fine-Tuning (SFT). Other methods include preference optimization (DPO, ORPO), distillation and [Reinforcement Learning (RL)](/docs/get-started/reinforcement-learning-rl-guide) (GRPO, GSPO), where an "agent" learns to make decisions by interacting with an environment and receiving **feedback** in the form of **rewards** or **penalties**.

With [Unsloth](https://github.com/unslothai/unsloth), you can fine-tune or do RL for free on Colab, Kaggle, or locally with just 3GB VRAM by using our [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks). By fine-tuning a pre-trained model on a dataset, you can:

* **Update + Learn New Knowledge**: Inject and learn new domain-specific information.
* **Customize Behavior**: Adjust the model’s tone, personality, or response style.
* **Optimize for Tasks**: Improve accuracy and relevance for specific use cases.

**Example fine-tuning or RL use-cases**:

* Enables LLMs to predict if a headline impacts a company positively or negatively.
* Can use historical customer interactions for more accurate and custom responses.
* Fine-tune LLM on legal texts for contract analysis, case law research, and compliance.

You can think of a fine-tuned model as a specialized agent designed to do specific tasks more effectively and efficiently. **Fine-tuning can replicate all of RAG's capabilities**, but not vice versa.

{% columns %}
{% column %}

#### :question:What is LoRA/QLoRA?

In LLMs, we have model weights. Llama 70B has 70 billion numbers. Instead of changing all 70B numbers, we instead add thin matrices A and B to each weight, and optimize those. This means we only optimize 1% of weights. LoRA is when the original model is 16-bit unquantized while QLoRA quantizes to 4-bit to save 75% memory.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-715b6260aae497f160d7f9a1019bcfa472675dcf%2Fimage%20(7)%20(1)%20(1).png?alt=media" alt=""><figcaption><p>Instead of optimizing Model Weights (yellow), we optimize 2 thin matrices A and B.</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

#### Fine-tuning misconceptions:

You may have heard that fine-tuning does not make a model learn new knowledge or RAG performs better than fine-tuning. That is **false**. You can train a specialized coding model with fine-tuning and RL while RAG can’t change the model’s weights and only augments what the model sees at inference time. Read more FAQ + misconceptions [here](https://unsloth.ai/docs/get-started/pages/HP82bIzgldwxWk3OSzVy#fine-tuning-vs.-rag-whats-the-difference):

{% content-ref url="/pages/HP82bIzgldwxWk3OSzVy" %}
[FAQ + Is Fine-tuning Right For Me?](/docs/get-started/fine-tuning-for-beginners/faq-+-is-fine-tuning-right-for-me)
{% endcontent-ref %}

> [**Introducing Unsloth Studio:** ](/docs/new/studio) Our new open-source web UI for training and running models. This means you can now fine-tune models with no-code and have observability and automatic dataset creation features.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FxV1PO5DbF3ksB51nE2Tw%2Fmore%20cropped%20ui%20for%20homepage.png?alt=media&amp;token=f75942c9-3d8d-4b59-8ba2-1a4a38de1b86" alt="" width="563"><figcaption></figcaption></figure></div>

## 2. Choose the Right Model + Method

If you're a beginner, it is best to start with a small instruct model like Llama 3.1 (8B) and experiment from there. You'll also need to decide between normal fine-tuning, RL, QLoRA or LoRA training:

* **Reinforcement Learning (RL)** is used when you need a model to excel at a specific behavior (e.g., tool-calling) using an environment and reward function rather than labeled data. We have several [notebook examples](/docs/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks), but for most use-cases, standard SFT is sufficient.
* **LoRA** is a parameter efficient training method that typically keeps the base model’s weights frozen and trains a small set of added low-rank adapter weights (in 16-bit precision).
* **QLoRA** combines LoRA with 4-bit precision to handle very large models with minimal resources.
* Unsloth also supports full fine-tuning (FFT) and pretraining, which require significantly more resources, but FFT is usually unnecessary. When done correctly, LoRA can match FFT.
* Unsloth **all types models**: [text-to-speech](/docs/basics/text-to-speech-tts-fine-tuning), [embedding](/docs/basics/embedding-finetuning), GRPO, RL, [vision](/docs/basics/vision-fine-tuning), multimodal and more.

{% hint style="info" %}
Research shows that **training and serving in the same precision** helps preserve accuracy. This means if you want to serve in 4-bit, train in 4-bit and vice versa.
{% endhint %}

We recommend starting with QLoRA, as it is one of the most accessible and effective methods for training models. Our [dynamic 4-bit](https://unsloth.ai/blog/dynamic-4bit) quants, the accuracy loss for QLoRA compared to LoRA is now largely recovered.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-cfc51c261e6d24df3aa967d9b9a482313465cbc1%2Fmodel%20name%20change.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

You can change the model name to whichever model you like by matching it with model's name on Hugging Face e.g. '`unsloth/llama-3.1-8b-unsloth-bnb-4bit`'.

We recommend starting with **Instruct models**, as they allow direct fine-tuning using conversational chat templates (ChatML, ShareGPT etc.) and require less data compared to **Base models** (which uses Alpaca, Vicuna etc). Learn more about the differences between [instruct and base models here](/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use#instruct-or-base-model).

* Model names ending in **`unsloth-bnb-4bit`** indicate they are [**Unsloth dynamic 4-bit**](https://unsloth.ai/blog/dynamic-4bit) **quants**. These models consume slightly more VRAM than standard BitsAndBytes 4-bit models but offer significantly higher accuracy.
* If a model name ends with just **`bnb-4bit`**, without "unsloth", it refers to a standard BitsAndBytes 4-bit quantization.
* Models with **no suffix** are in their original **16-bit or 8-bit formats**. While they are the original models from the official model creators, we sometimes include important fixes - such as chat template or tokenizer fixes. So it's recommended to use our versions when available.

There are other settings which you can toggle:

* **`max_seq_length = 2048`** – Controls context length. While Llama-3 supports 8192, we recommend 2048 for testing. Unsloth enables 4× longer context fine-tuning.
* **`dtype = None`** – Defaults to None; use `torch.float16` or `torch.bfloat16` for newer GPUs.
* **`load_in_4bit = True`** – Enables 4-bit quantization, reducing memory use 4× for fine-tuning. Disabling it enables LoRA 16-bit fine-tuning. You can also enable 16-bit LoRA with `load_in_16bit = True`
* To enable full fine-tuning (FFT), set `full_finetuning = True`. For 8-bit fine-tuning, set `load_in_8bit = True`.
* **Note:** Only one training method can be set to `True` at a time.

{% hint style="info" %}
A common mistake is jumping straight into full fine-tuning (FFT), which is compute-heavy. Start by testing with LoRA or QLoRA first, if it won’t work there, it almost certainly won’t work with FFT. And if LoRA fails, don’t assume FFT will magically fix it.
{% endhint %}

You can also do [Text-to-speech (TTS)](/docs/basics/text-to-speech-tts-fine-tuning), [reasoning (GRPO)](/docs/get-started/reinforcement-learning-rl-guide), [vision](/docs/basics/vision-fine-tuning), [RL](/docs/get-started/reinforcement-learning-rl-guide/preference-dpo-orpo-and-kto) (GRPO, DPO), [continued pretraining](/docs/basics/continued-pretraining), text completion and other training methodologies with Unsloth.

{% columns %}
{% column %}
Read our guide on choosing models:

{% content-ref url="/pages/BSShKhLoFNlGWO5cN8VJ" %}
[What Model Should I Use?](/docs/get-started/fine-tuning-llms-guide/what-model-should-i-use)
{% endcontent-ref %}
{% endcolumn %}

{% column %}
For inidivudal tutorials on models:

{% content-ref url="/pages/BAeSP6aOxvSeDUzCgKOK" %}
[Complete LLM Directory](/docs/models/tutorials)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

## 3. Your Dataset

For LLMs, datasets are collections of data that can be used to train our models. In order to be useful for training, text data needs to be in a format that can be tokenized.

* You will need to create a dataset usually with 2 columns - question and answer. The quality and amount will largely reflect the end result of your fine-tune so it's imperative to get this part right.
* You can [synthetically generate data](/docs/get-started/fine-tuning-llms-guide/datasets-guide#synthetic-data-generation) and structure your dataset (into QA pairs) using ChatGPT or local LLMs.
* You can also use our new Synthetic Dataset notebook which automatically parses documents (PDFs, videos etc.), generates QA pairs and auto cleans data using local models like Llama 3.2. [Access the notebook here.](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_\(3B\).ipynb)
* Fine-tuning can learn from an existing repository of documents and continuously expand its knowledge base, but just dumping data alone won’t work as well. For optimal results, curate a well-structured dataset, ideally as question-answer pairs. This enhances learning, understanding, and response accuracy.
* But, that's not always the case, e.g. if you are fine-tuning a LLM for code, just dumping all your code data can actually enable your model to yield significant performance improvements, even without structured formatting. So it really depends on your use case.

***Read more about creating your dataset:***

{% content-ref url="/pages/XgcpRfamZHmHnRHBnVE4" %}
[Datasets Guide](/docs/get-started/fine-tuning-llms-guide/datasets-guide)
{% endcontent-ref %}

For most of our notebook examples, we utilize the [Alpaca dataset](https://docs.unsloth.ai/basics/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-6.-alpaca-dataset) however other notebooks like Vision will use different datasets which may need images in the answer ouput as well.

### 4. Understand Training Hyperparameters

Learn how to choose the right [hyperparameters](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) using best practices from research and real-world experiments - and understand how each one affects your model's performance.

**For a complete guide on how hyperparameters affect training, see:**

{% content-ref url="/pages/y6obKRSk8TwyjIrCjuGE" %}
[Hyperparameters Guide](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide)
{% endcontent-ref %}

## 5. Install + Requirements

You can use Unsloth via two main ways, our free notebooks or locally.

### Unsloth Notebooks

We would recommend beginners to utilise our pre-made [notebooks](/docs/get-started/unsloth-notebooks) first as it's the easiest way to get started with guided steps. You can later export the notebooks to use locally.

Unsloth has step-by-step notebooks for [text-to-speech](/docs/basics/text-to-speech-tts-fine-tuning), [embedding](/docs/basics/embedding-finetuning), GRPO, RL, [vision](/docs/basics/vision-fine-tuning), multimodal, different use-cases and more.

### Local Installation

You can also install Unsloth locally via [Docker](/docs/get-started/install/docker) or `pip install unsloth` (with Linux, WSL or [Windows](/docs/get-started/install/windows-installation)). Also depending on the model you're using, you'll need enough VRAM and resources.

Installing Unsloth will require a Windows or Linux device. Once you install Unsloth, you can copy and paste our notebooks and use them in your own local environment. See:

{% columns %}
{% column %}
{% content-ref url="/pages/odJXZM9jv284RKqZ2Pna" %}
[Unsloth Requirements](/docs/get-started/fine-tuning-for-beginners/unsloth-requirements)
{% endcontent-ref %}
{% endcolumn %}

{% column %}
{% content-ref url="/pages/WbSfE0ITQYsNqERZwnbZ" %}
[Installation](/docs/get-started/install)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

## 6. Training + Evaluation

Once you have everything set, it's time to train! If something's not working, remember you can always change hyperparameters, your dataset etc.

You’ll see a log of numbers during training. This is the training loss, which shows how well the model is learning from your dataset. For many cases, a loss around 0.5 to 1.0 is a good sign, but it depends on your dataset and task. If the loss is not going down, you might need to adjust your settings. If the loss goes to 0, that could mean overfitting, so it's important to check validation too.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-feb9b0f5763d41cecaec9a3a9cd227ad918f0ca7%2Fimage.png?alt=media" alt="" width="375"><figcaption><p>The training loss will appear as numbers</p></figcaption></figure>

We generally recommend keeping the default settings unless you need longer training or larger batch sizes.

* **`per_device_train_batch_size = 2`** – Increase for better GPU utilization but beware of slower training due to padding. Instead, increase `gradient_accumulation_steps` for smoother training.
* **`gradient_accumulation_steps = 4`** – Simulates a larger batch size without increasing memory usage.
* **`max_steps = 60`** – Speeds up training. For full runs, replace with `num_train_epochs = 1` (1–3 epochs recommended to avoid overfitting).
* **`learning_rate = 2e-4`** – Lower for slower but more precise fine-tuning. Try values like `1e-4`, `5e-5`, or `2e-5`.

#### Evaluation

In order to evaluate, you could do manually evaluation by just chatting with the model and see if it's to your liking. You can also enable evaluation for Unsloth, but keep in mind it can be time-consuming depending on the dataset size. To speed up evaluation you can: reduce the evaluation dataset size or set `evaluation_steps = 100`.

For testing, you can also take 20% of your training data and use that for testing. If you already used all of the training data, then you have to manually evaluate it. You can also use automatic eval tools but keep in mind that automated tools may not perfectly align with your evaluation criteria.

## 7. Running + Deploying the model

Now let's run the model after we completed the training process! You can edit the yellow underlined part! In fact, because we created a multi turn chatbot, we can now also call the model as if it saw some conversations in the past like below:

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-f2d5f23fa62ec89e06bf20fea433f9a1e42a2fe3%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-cdf5d779635901dce7793df92531dbf3caf0fb0a%2Fimage%20(47).png?alt=media" alt=""><figcaption></figcaption></figure></div>

Reminder Unsloth itself provides **2x faster inference** natively as well, so always do not forget to call `FastLanguageModel.for_inference(model)`. If you want the model to output longer responses, set `max_new_tokens = 128` to some larger number like 256 or 1024. Notice you will have to wait longer for the result as well!

### Saving + Deployment

For saving and deploying your model in desired inference engines like Ollama, vLLM, Open WebUI, you will need to use the LoRA adapter on top of the base model. We have designated guides for each framework:

{% content-ref url="/pages/gEugERiAw2ztDNt98JVR" %}
[Inference & Deployment](/docs/basics/inference-and-deployment)
{% endcontent-ref %}

{% columns %}
{% column %}
If you’re running inference on a single device (like a laptop or Mac), use llama.cpp to convert to GGUF format to use in Ollama, llama.cpp, LM Studio etc:

{% content-ref url="/pages/T7ZPf3SNAwDykZNgXptE" %}
[GGUF & llama.cpp](/docs/basics/inference-and-deployment/saving-to-gguf)
{% endcontent-ref %}
{% endcolumn %}

{% column %}
If you’re deploying an LLM for enterprise or multi-user inference for FP8, AWQ, use vLLM:

{% content-ref url="/pages/fhJtaLFFXVsGnbMUiACo" %}
[vLLM](/docs/basics/inference-and-deployment/vllm-guide)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

We can now save the fine-tuned model as a small 100MB file called a LoRA adapter like below. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a Hugging Face [token](https://huggingface.co/settings/tokens) and add your token!

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-8c577103f7c4fe883cabaf35c8437307c6501686%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1a1be852ca551240bdce47cf99e6ccd7d31c1326%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure></div>

After saving the model, we can again use Unsloth to run the model itself! Use `FastLanguageModel` again to call it for inference!

## 8. We're done!

You've successfully fine-tuned a language model and exported it to your desired inference engine with Unsloth!

To learn more about fine-tuning tips and tricks, head over to our blogs which provide tremendous and educational value: <https://unsloth.ai/blog/>

If you need any help on fine-tuning, you can also join our Discord server [here](https://discord.gg/unsloth) or [Reddit r/unsloth](https://www.reddit.com/r/unsloth/). Thanks for reading and hopefully this was helpful!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-69482ba90d417f7bf98dddaf83795cdd3eb20efc%2Fsloth%20sparkling%20square.png?alt=media" alt="" width="188"><figcaption></figcaption></figure>


# Datasets Guide

Learn how to create & prepare a dataset for fine-tuning.

## What is a Dataset?

For LLMs, datasets are collections of data that can be used to train our models. In order to be useful for training, text data needs to be in a format that can be tokenized. You'll also learn how to [use datasets inside of Unsloth](#applying-chat-templates-with-unsloth).

One of the key parts of creating a dataset is your [chat template](/docs/basics/chat-templates) and how you are going to design it. Tokenization is also important as it breaks text into tokens, which can be words, sub-words, or characters so LLMs can process it effectively. These tokens are then turned into embeddings and are adjusted to help the model understand the meaning and context.

### Data Format

To enable the process of tokenization, datasets need to be in a format that can be read by a tokenizer.

<table data-full-width="false"><thead><tr><th>Format</th><th>Description</th><th>Training Type</th></tr></thead><tbody><tr><td>Raw Corpus</td><td>Raw text from a source such as a website, book, or article.</td><td>Continued Pretraining (CPT)</td></tr><tr><td>Instruct</td><td>Instructions for the model to follow and an example of the output to aim for.</td><td>Supervised fine-tuning (SFT)</td></tr><tr><td>Conversation</td><td>Multiple-turn conversation between a user and an AI assistant.</td><td>Supervised fine-tuning (SFT)</td></tr><tr><td>RLHF</td><td>Conversation between a user and an AI assistant, with the assistant's responses being ranked by a script, another model or human evaluator.</td><td>Reinforcement Learning (RL)</td></tr></tbody></table>

{% hint style="info" %}
It's worth noting that different styles of format exist for each of these types.
{% endhint %}

## Getting Started

Before we format our data, we want to identify the following:

{% stepper %}
{% step %} <mark style="color:green;">Purpose of dataset</mark>

Knowing the purpose of the dataset will help us determine what data we need and format to use.

The purpose could be, adapting a model to a new task such as summarization or improving a model's ability to role-play a specific character. For example:

* Chat-based dialogues (Q\&A, learn a new language, customer support, conversations).
* Structured tasks ([classification](https://colab.research.google.com/github/timothelaborie/text_classification_scripts/blob/main/unsloth_classification.ipynb), summarization, generation tasks).
* Domain-specific data (medical, finance, technical).
  {% endstep %}

{% step %} <mark style="color:green;">Style of output</mark>

The style of output will let us know what sources of data we will use to reach our desired output.

For example, the type of output you want to achieve could be JSON, HTML, text or code. Or perhaps you want it to be Spanish, English or German etc.
{% endstep %}

{% step %} <mark style="color:green;">Data source</mark>

When we know the purpose and style of the data we need, we need to analyze the quality and [quantity](#how-big-should-my-dataset-be) of the data. Hugging Face and Wikipedia are great sources of datasets and Wikipedia is especially useful if you are looking to train a model to learn a language.

The Source of data can be a CSV file, PDF or even a website. You can also [synthetically generate](#synthetic-data-generation) data but extra care is required to make sure each example is high quality and relevant.
{% endstep %}
{% endstepper %}

{% hint style="success" %}
One of the best ways to create a better dataset is by combining it with a more generalized dataset from Hugging Face like ShareGPT to make your model smarter and diverse. You could also add [synthetically generated data](#synthetic-data-generation).
{% endhint %}

## 🦥 Unsloth Data Recipes

[Unsloth Data Recipes](/docs/new/studio/data-recipe) lets you upload documents like PDFs or CSVs files and transforms them into useable datasets. Create and edit datasets visually via a graph-node workflow.

The recipes page is the main entry point. Recipes are stored locally in the browser, so you come back to saved work later. From here, you can create a blank recipe or open a guided learning recipe.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fc5m3JX1kUA3UwmdcJcxH%2FArea.gif?alt=media&amp;token=33bbd908-7d6c-456a-bc58-ce495c0adca1" alt=""><figcaption></figcaption></figure></div>

Data Recipes follows the same basic path. You open the recipes page, create or pick a recipe, build the workflow in the editor, validate it run a preview, then run the full dataset once the output looks right. Add seed data and generation blocks, validate the workflow, preview sample output, then run a full dataset build.

At a glance a usual workflow should look like this:

1. Open the recipes page.
2. Create a new recipe or open an existing one.
3. Add blocks to define your dataset workflow.
4. Click **Validate** to catch configuration issues early.
5. Run a preview to inspect sample rows quickly.
6. Run a full dataset build when the recipe is ready.
7. Review progress and output live in graph or in **Executions** view for mode details.
8. Select the resulting dataset in Unsloth and fine tune a model. Read more:

{% content-ref url="/pages/m9k4PLFmjpsAP6LsQt7u" %}
[Data Recipes](/docs/new/studio/data-recipe)
{% endcontent-ref %}

## Formatting the Data

When we have identified the relevant criteria, and collected the necessary data, we can then format our data into a machine readable format that is ready for training.

### Common Data Formats for LLM Training

For [**continued pretraining**](/docs/basics/continued-pretraining), we use raw text format without specific structure:

```json
  "text": "Pasta carbonara is a traditional Roman pasta dish. The sauce is made by mixing raw eggs with grated Pecorino Romano cheese and black pepper. The hot pasta is then tossed with crispy guanciale (cured pork cheek) and the egg mixture, creating a creamy sauce from the residual heat. Despite popular belief, authentic carbonara never contains cream or garlic. The dish likely originated in Rome in the mid-20th century, though its exact origins are debated..."
```

This format preserves natural language flow and allows the model to learn from continuous text.

If we are adapting a model to a new task, and intend for the model to output text in a single turn based on a specific set of instructions, we can use **Instruction** format in [Alpaca style](https://docs.unsloth.ai/basics/tutorial-how-to-finetune-llama-3-and-use-in-ollama#id-6.-alpaca-dataset)

```json
"Instruction": "Task we want the model to perform."

"Input": "Optional, but useful, it will essentially be the user's query."

"Output": "The expected result of the task and the output of the model."
```

When we want multiple turns of conversation we can use the ShareGPT format:

```json
{
  "conversations": [
    {
      "from": "human",
      "value": "Can you help me make pasta carbonara?"
    },
    {
      "from": "gpt",
      "value": "Would you like the traditional Roman recipe, or a simpler version?"
    },
    {
      "from": "human",
      "value": "The traditional version please"
    },
    {
      "from": "gpt",
      "value": "The authentic Roman carbonara uses just a few ingredients: pasta, guanciale, eggs, Pecorino Romano, and black pepper. Would you like the detailed recipe?"
    }
  ]
}
```

The template format uses the "from"/"value" attribute keys and messages alternates between `human`and `gpt`, allowing for natural dialogue flow.

The other common format is OpenAI's ChatML format and is what Hugging Face defaults to. This is probably the most used format, and alternates between `user` and `assistant`

```
{
  "messages": [
    {
      "role": "user",
      "content": "What is 1+1?"
    },
    {
      "role": "assistant",
      "content": "It's 2!"
    },
  ]
}
```

### Applying Chat Templates with Unsloth

For datasets that usually follow the common chatml format, the process of preparing the dataset for training or finetuning, consists of four simple steps:

* Check the chat templates that Unsloth currently supports:\\

  ```
  from unsloth.chat_templates import CHAT_TEMPLATES
  print(list(CHAT_TEMPLATES.keys()))
  ```

  \
  This will print out the list of templates currently supported by Unsloth. Here is an example output:\\

  ```
  ['unsloth', 'zephyr', 'chatml', 'mistral', 'llama', 'vicuna', 'vicuna_old', 'vicuna old', 'alpaca', 'gemma', 'gemma_chatml', 'gemma2', 'gemma2_chatml', 'llama-3', 'llama3', 'phi-3', 'phi-35', 'phi-3.5', 'llama-3.1', 'llama-31', 'llama-3.2', 'llama-3.3', 'llama-32', 'llama-33', 'qwen-2.5', 'qwen-25', 'qwen25', 'qwen2.5', 'phi-4', 'gemma-3', 'gemma3']
  ```

  \\
* Use `get_chat_template` to apply the right chat template to your tokenizer:\\

  ```
  from unsloth.chat_templates import get_chat_template

  tokenizer = get_chat_template(
      tokenizer,
      chat_template = "gemma-3", # change this to the right chat_template name
  )
  ```

  \\
* Define your formatting function. Here's an example:\\

  ```
  def formatting_prompts_func(examples):
     convos = examples["conversations"]
     texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]
     return { "text" : texts, }
  ```

  \
  \
  This function loops through your dataset applying the chat template you defined to each sample.\\
* Finally, let's load the dataset and apply the required modifications to our dataset: \\

  ```
  # Import and load dataset
  from datasets import load_dataset
  dataset = load_dataset("repo_name/dataset_name", split = "train")

  # Apply the formatting function to your dataset using the map method
  dataset = dataset.map(formatting_prompts_func, batched = True,)
  ```

  \
  If your dataset uses the ShareGPT format with "from"/"value" keys instead of the ChatML "role"/"content" format, you can use the `standardize_sharegpt` function to convert it first. The revised code will now look as follows:\
  \\

  ```
  # Import dataset
  from datasets import load_dataset
  dataset = load_dataset("mlabonne/FineTome-100k", split = "train")

  # Convert your dataset to the "role"/"content" format if necessary
  from unsloth.chat_templates import standardize_sharegpt
  dataset = standardize_sharegpt(dataset)

  # Apply the formatting function to your dataset using the map method
  dataset = dataset.map(formatting_prompts_func, batched = True,)
  ```

### Formatting Data Q\&A

<mark style="color:green;">**Q:**</mark> How can I use the Alpaca instruct format?

<mark style="color:green;">**A:**</mark> If your dataset is already formatted in the Alpaca format, then follow the formatting steps as shown in the Llama3.1 [notebook ](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_\(8B\)-Alpaca.ipynb#scrollTo=LjY75GoYUCB8). If you need to convert your data to the Alpaca format, one approach is to create a Python script to process your raw data. If you're working on a summarization task, you can use a local LLM to generate instructions and outputs for each example.

<mark style="color:green;">**Q:**</mark> Should I always use the standardize\_sharegpt method?

<mark style="color:green;">**A:**</mark> Only use the standardize\_sharegpt method if your target dataset is formatted in the sharegpt format, but your model expect a ChatML format instead.

\ <mark style="color:green;">**Q:**</mark> Why not use the apply\_chat\_template function that comes with the tokenizer.

<mark style="color:green;">**A:**</mark> The `chat_template` attribute when a model is first uploaded by the original model owners sometimes contains errors and may take time to be updated. In contrast, at Unsloth, we thoroughly check and fix any errors in the `chat_template` for every model when we upload the quantized versions to our repositories. Additionally, our `get_chat_template` and `apply_chat_template` methods offer advanced data manipulation features, which are fully documented on our Chat Templates documentation [page](https://docs.unsloth.ai/basics/chat-templates).

<mark style="color:green;">**Q:**</mark> What if my template is not currently supported by Unsloth?

<mark style="color:green;">**A:**</mark> Submit a feature request on the unsloth github issues [forum](https://github.com/unslothai/unsloth). As a temporary workaround, you could also use the tokenizer's own apply\_chat\_template function until your feature request is approved and merged.

## Synthetic Data Generation

You can also use any local LLM like Llama 3.3 (70B) or OpenAI's GPT 4.5 to generate synthetic data. Generally, it is better to use a bigger like Llama 3.3 (70B) to ensure the highest quality outputs. You can directly use inference engines like vLLM, Ollama or llama.cpp to generate synthetic data but it will require some manual work to collect it and prompt for more data. There's 3 goals for synthetic data:

* Produce entirely new data - either from scratch or from your existing dataset
* Diversify your dataset so your model does not [overfit](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide#avoiding-overfitting-and-underfitting) and become too specific
* Augment existing data e.g. automatically structure your dataset in the correct chosen format

### Using Unsloth for synthetic data

You can easily upload any unstructured or structured data into Unsloth Studio's [Data Recipes](/docs/new/studio/data-recipe) and it will automatically convert it into a useable / synthetic dataset. More details in [our guide](/docs/new/studio/data-recipe).

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQ6e19jESrJg0VjHnX58c%2Fdata%20recipes%20final.png?alt=media&amp;token=8d74e453-815d-4790-83d1-76d0bc80a3ce" alt="" width="563"><figcaption></figcaption></figure></div>

### Using a local LLM or ChatGPT for synthetic data

Your goal is to prompt the model to generate and process QA data that is in your specified format. The model will need to learn the structure that you provided and also the context so ensure you at least have 10 examples of data already. Examples prompts:

* **Prompt for generating more dialogue on an existing dataset**:

  <pre data-overflow="wrap"><code><strong>Using the dataset example I provided, follow the structure and generate conversations based on the examples.
  </strong></code></pre>
* **Prompt if you no have dataset**:

  {% code overflow="wrap" %}

  ```
  Create 10 examples of product reviews for Coca-Coca classified as either positive, negative, or neutral.
  ```

  {% endcode %}
* **Prompt for a dataset without formatting**:

  {% code overflow="wrap" %}

  ```
  Structure my dataset so it is in a QA ChatML format for fine-tuning. Then generate 5 synthetic data examples with the same topic and format.
  ```

  {% endcode %}

It is recommended to check the quality of generated data to remove or improve on irrelevant or poor-quality responses. Depending on your dataset it may also have to be balanced in many areas so your model does not overfit. You can then feed this cleaned dataset back into your LLM to regenerate data, now with even more guidance.

## Dataset FAQ + Tips

### How big should my dataset be?

We generally recommend using a bare minimum of at least 100 rows of data for fine-tuning to achieve reasonable results. For optimal performance, a dataset with over 1,000 rows is preferable, and in this case, more data usually leads to better outcomes. If your dataset is too small you can also add synthetic data or add a dataset from Hugging Face to diversify it. However, the effectiveness of your fine-tuned model depends heavily on the quality of the dataset, so be sure to thoroughly clean and prepare your data.

### How should I structure my dataset if I want to fine-tune a reasoning model?

If you want to fine-tune a model that already has reasoning capabilities like the distilled versions of DeepSeek-R1 (e.g. DeepSeek-R1-Distill-Llama-8B), you will need to still follow question/task and answer pairs however, for your answer you will need to change the answer so it includes reasoning/chain-of-thought process and the steps it took to derive the answer.\
\
For a model that does not have reasoning and you want to train it so that it later encompasses reasoning capabilities, you will need to utilize a standard dataset but this time without reasoning in its answers. This is training process is known as [Reinforcement Learning and GRPO](/docs/get-started/reinforcement-learning-rl-guide).

### Multiple datasets

If you have multiple datasets for fine-tuning, you can either:

* Standardize the format of all datasets, combine them into a single dataset, and fine-tune on this unified dataset.
* Use the [Multiple Datasets](https://colab.research.google.com/drive/1njCCbE1YVal9xC83hjdo2hiGItpY_D6t?usp=sharing) notebook to fine-tune on multiple datasets directly.

### Can I fine-tune the same model multiple times?

You can fine-tune an already fine-tuned model multiple times, but it's best to combine all the datasets and perform the fine-tuning in a single process instead. Training an already fine-tuned model can potentially alter the quality and knowledge acquired during the previous fine-tuning process.

## Using Datasets in Unsloth

### Alpaca Dataset

See an example of using the Alpaca dataset inside of Unsloth on Google Colab:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1d66d8714e44d90513dd87b9356eec67886ab3f7%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

We will now use the Alpaca Dataset created by calling GPT-4 itself. It is a list of 52,000 instructions and outputs which was very popular when Llama-1 was released, since it made finetuning a base LLM be competitive with ChatGPT itself.

You can access the GPT4 version of the Alpaca dataset [here](https://huggingface.co/datasets/vicgalle/alpaca-gpt4.). Below shows some examples of the dataset:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-0dde50e386e7b245d3e8a57e10a4a81755b3769a%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

You can see there are 3 columns in each row - an instruction, and input and an output. We essentially combine each row into 1 large prompt like below. We then use this to finetune the language model, and this made it very similar to ChatGPT. We call this process **supervised instruction finetuning**.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-8b3663c5d80adcb935ff77661500f08e13c9af2d%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

### Multiple columns for finetuning

But a big issue is for ChatGPT style assistants, we only allow 1 instruction / 1 prompt, and not multiple columns / inputs. For example in ChatGPT, you can see we must submit 1 prompt, and not multiple prompts.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-d90162c2685ced871f4151369aadcaee40a9c54f%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

This essentially means we have to "merge" multiple columns into 1 large prompt for finetuning to actually function!

For example the very famous Titanic dataset has many many columns. Your job was to predict whether a passenger has survived or died based on their age, passenger class, fare price etc. We can't simply pass this into ChatGPT, but rather, we have to "merge" this information into 1 large prompt.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a2df04874bfc879182cb66c789341d49700227ea%2FMerge.png?alt=media" alt=""><figcaption></figcaption></figure>

For example, if we ask ChatGPT with our "merged" single prompt which includes all the information for that passenger, we can then ask it to guess or predict whether the passenger has died or survived.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-b3da2b36afe37469cd3962f37186e758871864a5%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Other finetuning libraries require you to manually prepare your dataset for finetuning, by merging all your columns into 1 prompt. In Unsloth, we simply provide the function called `to_sharegpt` which does this in 1 go!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-62b94dc44f2e343020d31de575f52eb22be4b0fc%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Now this is a bit more complicated, since we allow a lot of customization, but there are a few points:

* You must enclose all columns in curly braces `{}`. These are the column names in the actual CSV / Excel file.
* Optional text components must be enclosed in `[[]]`. For example if the column "input" is empty, the merging function will not show the text and skip this. This is useful for datasets with missing values.
* Select the output or target / prediction column in `output_column_name`. For the Alpaca dataset, this will be `output`.

For example in the Titanic dataset, we can create a large merged prompt format like below, where each column / piece of text becomes optional.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e6228cf6e5c0bb4e4b45e6f3e045910d567c33d2%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

For example, pretend the dataset looks like this with a lot of missing data:

| Embarked | Age | Fare |
| -------- | --- | ---- |
| S        | 23  |      |
|          | 18  | 7.25 |

Then, we do not want the result to be:

1. The passenger embarked from S. Their age is 23. Their fare is **EMPTY**.
2. The passenger embarked from **EMPTY**. Their age is 18. Their fare is $7.25.

Instead by optionally enclosing columns using `[[]]`, we can exclude this information entirely.

1. \[\[The passenger embarked from S.]] \[\[Their age is 23.]] \[\[Their fare is **EMPTY**.]]
2. \[\[The passenger embarked from **EMPTY**.]] \[\[Their age is 18.]] \[\[Their fare is $7.25.]]

becomes:

1. The passenger embarked from S. Their age is 23.
2. Their age is 18. Their fare is $7.25.

### Multi turn conversations

An issue if you didn't notice is the Alpaca dataset is single turn, whilst remember using ChatGPT was interactive and you can talk to it in multiple turns. For example, the left is what we want, but the right which is the Alpaca dataset only provides singular conversations. We want the finetuned language model to somehow learn how to do multi turn conversations just like ChatGPT.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2a65cd74ddd03a6bcbbc9827d9d034e4879a8e6a%2Fdiff.png?alt=media" alt=""><figcaption></figcaption></figure>

So we introduced the `conversation_extension` parameter, which essentially selects some random rows in your single turn dataset, and merges them into 1 conversation! For example, if you set it to 3, we randomly select 3 rows and merge them into 1! Setting them too long can make training slower, but could make your chatbot and final finetune much better!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2b1b3494b260f1102942d86143a885225c6a06f2%2Fcombine.png?alt=media" alt=""><figcaption></figcaption></figure>

Then set `output_column_name` to the prediction / output column. For the Alpaca dataset dataset, it would be the output column.

We then use the `standardize_sharegpt` function to just make the dataset in a correct format for finetuning! Always call this!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-7bf83bf802191bda9e417bbe45afa181e7f24f38%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## Vision Fine-tuning

The dataset for fine-tuning a vision or multimodal model also includes image inputs. For example, the [Llama 3.2 Vision Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(11B\)-Vision.ipynb#scrollTo=vITh0KVJ10qX) uses a radiography case to show how AI can help medical professionals analyze X-rays, CT scans, and ultrasounds more efficiently.

We'll be using a sampled version of the ROCO radiography dataset. You can access the dataset [here](https://www.google.com/url?q=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Funsloth%2FRadiology_mini). The dataset includes X-rays, CT scans and ultrasounds showcasing medical conditions and diseases. Each image has a caption written by experts describing it. The goal is to finetune a VLM to make it a useful analysis tool for medical professionals.

Let's take a look at the dataset, and check what the 1st example shows:

```
Dataset({
    features: ['image', 'image_id', 'caption', 'cui'],
    num_rows: 1978
})
```

| Image                                                                                                                                                                                                                                  | Caption                                                                                                                                       |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| <img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-97d4489827403bd4795494f33d01a10979788c30%2Fxray.png?alt=media" alt="" data-size="original"> | Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows). |

To format the dataset, all vision finetuning tasks should be formatted as follows:

```python
[
{ "role": "user",
  "content": [{"type": "text",  "text": instruction}, {"type": "image", "image": image} ]
},
{ "role": "assistant",
  "content": [{"type": "text",  "text": answer} ]
},
]
```

We will craft an custom instruction asking the VLM to be an expert radiographer. Notice also instead of just 1 instruction, you can add multiple turns to make it a dynamic conversation.

```notebook-python
instruction = "You are an expert radiographer. Describe accurately what you see in this image."

def convert_to_conversation(sample):
    conversation = [
        { "role": "user",
          "content" : [
            {"type" : "text",  "text"  : instruction},
            {"type" : "image", "image" : sample["image"]} ]
        },
        { "role" : "assistant",
          "content" : [
            {"type" : "text",  "text"  : sample["caption"]} ]
        },
    ]
    return { "messages" : conversation }
pass
```

Let's convert the dataset into the "correct" format for finetuning:

```notebook-python
converted_dataset = [convert_to_conversation(sample) for sample in dataset]
```

The first example is now structured like below:

```notebook-python
converted_dataset[0]
```

{% code overflow="wrap" %}

```
{'messages': [{'role': 'user',
   'content': [{'type': 'text',
     'text': 'You are an expert radiographer. Describe accurately what you see in this image.'},
    {'type': 'image',
     'image': <PIL.PngImagePlugin.PngImageFile image mode=L size=657x442>}]},
  {'role': 'assistant',
   'content': [{'type': 'text',
     'text': 'Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows).'}]}]}
```

{% endcode %}

Before we do any finetuning, maybe the vision model already knows how to analyse the images? Let's check if this is the case!

```notebook-python
FastVisionModel.for_inference(model) # Enable for inference!

image = dataset[0]["image"]
instruction = "You are an expert radiographer. Describe accurately what you see in this image."

messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": instruction}
    ]}
]
input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True)
inputs = tokenizer(
    image,
    input_text,
    add_special_tokens = False,
    return_tensors = "pt",
).to("cuda")

from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer, skip_prompt = True)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,
                   use_cache = True, temperature = 1.5, min_p = 0.1)
```

And the result:

```
This radiograph appears to be a panoramic view of the upper and lower dentition, specifically an Orthopantomogram (OPG).

* The panoramic radiograph demonstrates normal dental structures.
* There is an abnormal area on the upper right, represented by an area of radiolucent bone, corresponding to the antrum.

**Key Observations**

* The bone between the left upper teeth is relatively radiopaque.
* There are two large arrows above the image, suggesting the need for a closer examination of this area. One of the arrows is in a left-sided position, and the other is in the right-sided position. However, only
```

For more details, view our dataset section in the [notebook here](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(11B\)-Vision.ipynb#scrollTo=vITh0KVJ10qX).


# LoRA fine-tuning Hyperparameters Guide

Learn step-by-step the best LLM fine-tuning settings - LoRA rank & alpha, epochs, batch size + gradient accumulation, QLoRA vs. LoRA, target modules, and more.

LoRA hyperparameters are tunable settings that govern how Low-Rank Adaptation [fine-tunes](/docs/get-started/fine-tuning-llms-guide) LLMs. With many choices (e.g., learning rate and epochs) and countless combinations, picking the right values is key to accuracy, stability, quality, and fewer hallucinations. Done well, **LoRA can match full fine-tuning performance** while using 4× less VRAM.

You'll learn the best practices for these parameters, based on insights from hundreds of research papers and experiments, and see how they impact the model. **While we recommend using Unsloth's defaults**, understanding these concepts will give you full control.\
\
The goal is to change hyperparameter numbers to increase accuracy while counteracting [**overfitting or underfitting**](#overfitting-poor-generalization-too-specialized). Overfitting occurs when the model memorizes the training data, harming its ability to generalize to new, unseen inputs. The objective is a model that generalizes well, not one that simply memorizes.

{% columns %}
{% column %}

#### :question:But what is LoRA?

In LLMs, we have model weights. Llama 70B has 70 billion numbers. Instead of changing all 70b numbers, we instead add thin matrices A and B to each weight, and optimize those. This means we only optimize 1% of weights.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-715b6260aae497f160d7f9a1019bcfa472675dcf%2Fimage%20(7)%20(1)%20(1).png?alt=media" alt=""><figcaption><p>Instead of optimizing Model Weights (yellow), we optimize 2 thin matrices A and B.</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

## :1234: Key Fine-tuning Hyperparameters

### **Learning Rate**

Defines how much the model’s weights are adjusted during each training step.

* **Higher Learning Rates**: Lead to faster initial convergence but can cause training to become unstable or fail to find an optimal minimum if set too high.
* **Lower Learning Rates**: Result in more stable and precise training but may require more epochs to converge, increasing overall training time. While low learning rates are often thought to cause underfitting, they actually can lead to **overfitting** or even prevent the model from learning.
* **Typical Range**: `2e-4` (0.0002) to `5e-6` (0.000005).\
  :green\_square: ***For normal LoRA/QLoRA Fine-tuning***, *we recommend* **`2e-4`** *as a starting point.*\
  :blue\_square: ***For Reinforcement Learning** (DPO, GRPO etc.), we recommend* **`5e-6` .**\
  :white\_large\_square: ***For Full Fine-tuning,** lower learning rates are generally more appropriate.*

### **Epochs**

The number of times the model sees the full training dataset.

* **More Epochs:** Can help the model learn better, but a high number can cause it to **memorize the training data**, hurting its performance on new tasks.
* **Fewer Epochs:** Reduces training time and can prevent overfitting, but may result in an undertrained model if the number is insufficient for the model to learn the dataset's underlying patterns.
* **Recommended:** 1-3 epochs. For most instruction-based datasets, training for more than 3 epochs offers diminishing returns and increases the risk of overfitting.

### **LoRA or QLoRA**

LoRA uses 16-bit precision, while QLoRA is a 4-bit fine-tuning method.

* **LoRA:** 16-bit fine-tuning. It's slightly faster and slightly more accurate, but consumes significantly more VRAM (4× more than QLoRA). Recommended for 16-bit environments and scenarios where maximum accuracy is required.
* **QLoRA:** 4-bit fine-tuning. Slightly slower and marginally less accurate, but uses much less VRAM (4× less).\
  :sloth: *70B LLaMA fits in <48GB VRAM with QLoRA in Unsloth -* [*more details here*](https://unsloth.ai/blog/llama3-3)*.*

### Hyperparameters & Recommendations:

<table><thead><tr><th width="154.39678955078125">Hyperparameter</th><th width="383.6192626953125">Function</th><th>Recommended Settings</th></tr></thead><tbody><tr><td><strong>LoRA Rank</strong> (<code>r</code>)</td><td>Controls the number of trainable parameters in the LoRA adapter matrices. A higher rank increases model capacity but also memory usage.</td><td>8, 16, 32, 64, 128<br><br>Choose 16 or 32</td></tr><tr><td><strong>LoRA Alpha</strong> (<code>lora_alpha</code>)</td><td>Scales the strength of the fine-tuned adjustments in relation to the rank (<code>r</code>).</td><td><code>r</code> (standard) or <code>r * 2</code> (common heuristic). <a href="#lora-alpha-and-rank-relationship">More details here</a>.</td></tr><tr><td><strong>LoRA Dropout</strong></td><td>A regularization technique that randomly sets a fraction of LoRA activations to zero during training to prevent overfitting. <strong>Not that useful</strong>, so we default set it to 0.</td><td>0 (default) to 0.1</td></tr><tr><td><strong>Weight Decay</strong></td><td>A regularization term that penalizes large weights to prevent overfitting and improve generalization. Don't use too large numbers!</td><td>0.01 (recommended) - 0.1</td></tr><tr><td><strong>Warmup Steps</strong></td><td>Gradually increases the learning rate at the start of training.</td><td>5-10% of total steps</td></tr><tr><td><strong>Scheduler Type</strong></td><td>Adjusts the learning rate dynamically during training.</td><td><code>linear</code> or <code>cosine</code></td></tr><tr><td><strong>Seed (<code>random_state</code>)</strong></td><td>A fixed number to ensure reproducibility of results.</td><td>Any integer (e.g., <code>42</code>, <code>3407</code>)</td></tr><tr><td><strong>Target Modules</strong></td><td><p>Specify which parts of the model you want to apply LoRA adapters to — either the attention, the MLP, or both.</p><p><br>Attention: <code>q_proj, k_proj, v_proj, o_proj</code><br><br>MLP: <code>gate_proj, up_proj, down_proj</code></p></td><td>Recommended to target all major linear layers: <code>q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj</code>.</td></tr></tbody></table>

## :deciduous\_tree: Gradient Accumulation and Batch Size equivalency

### Effective Batch Size

Correctly configuring your batch size is critical for balancing training stability with your GPU's VRAM limitations. This is managed by two parameters whose product is the **Effective Batch Size**.\
\
**Effective Batch Size** = `batch_size * gradient_accumulation_steps`

* A **larger Effective Batch Size** generally leads to smoother, more stable training.
* A **smaller Effective Batch Size** may introduce more variance.

While every task is different, the following configuration provides a great starting point for achieving a stable **Effective Batch Size** of 16, which works well for most fine-tuning tasks on modern GPUs.

| Parameter                                                 | Description                                                                                                                                                                                                                                                                     | Recommended Setting                             |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| **Batch Size** (`batch_size`)                             | <p>The number of samples processed in a single forward/backward pass on one GPU.<br><br><strong>Primary Driver of VRAM Usage</strong>. Higher values can improve hardware utilization and speed up training, but only if they fit in memory.</p>                                | 2                                               |
| **Gradient Accumulation** (`gradient_accumulation_steps`) | <p>The number of micro-batches to process before performing a single model weight update.<br><br><strong>Primary Driver of Training Time.</strong> Allows simulation of a larger <code>batch\_size</code> to conserve VRAM. Higher values increase training time per epoch.</p> | 8                                               |
| **Effective Batch Size** (Calculated)                     | The true batch size used for each gradient update. It directly influences training stability, quality, and final model performance.                                                                                                                                             | <p>4 to 16<br>Recommended: 16 (from 2 \* 8)</p> |

### The VRAM & Performance Trade-off

Assume you want 32 samples of data per training step. Then you can use any of the following configurations:

* `batch_size = 32, gradient_accumulation_steps = 1`
* `batch_size = 16, gradient_accumulation_steps = 2`
* `batch_size = 8, gradient_accumulation_steps = 4`
* `batch_size = 4, gradient_accumulation_steps = 8`
* `batch_size = 2, gradient_accumulation_steps = 16`
* `batch_size = 1, gradient_accumulation_steps = 32`

While all of these are equivalent for the model's weight updates, they have vastly different hardware requirements.

The first configuration (`batch_size = 32`) uses the **most VRAM** and will likely fail on most GPUs. The last configuration (`batch_size = 1`) uses the **least VRAM,** but at the cost of slightly slower trainin&#x67;**.** To avoid OOM (out of memory) errors, always prefer to set a smaller `batch_size` and increase `gradient_accumulation_steps` to reach your target **Effective Batch Size**.

### :sloth: Unsloth Gradient Accumulation Fix

Gradient accumulation and batch sizes <mark style="color:green;">**are now fully equivalent in Unsloth**</mark> due to our bug fixes for gradient accumulation. We have implemented specific bug fixes for gradient accumulation that resolve a common issue where the two methods did not produce the same results. This was a known challenge in the wider community, but for Unsloth users, the two methods are now interchangeable.

[Read our blog post](https://unsloth.ai/blog/gradient) for more details.

Prior to our fixes, combinations of `batch_size` and `gradient_accumulation_steps` that yielded the same **Effective Batch Size** (i.e., `batch_size × gradient_accumulation_steps = 16`) did not result in equivalent training behavior. For example, configurations like `b1/g16`, `b2/g8`, `b4/g4`, `b8/g2`, and `b16/g1` all have an **Effective Batch Size** of 16, but as shown in the graph, the loss curves did not align when using standard gradient accumulation:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-66eb907fd9ce38ab29dacef82794d0525057aeb4%2FBefore_-_Standard_gradient_accumulation_UQOFkUggudXuV9dzrh8MA.svg?alt=media" alt=""><figcaption><p>(Before - Standard Gradient Accumulation)</p></figcaption></figure>

After applying our fixes, the loss curves now align correctly, regardless of how the **Effective Batch Size** of 16 is achieved:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-61f7c60412a2a39584f75cce5dca41e3e35eb7f2%2FAfter_-_Unsloth_gradient_accumulation_6Y4pJdJF0vruzradUpymY.svg?alt=media" alt=""><figcaption><p>(After - 🦥 <mark style="color:green;">Unsloth Gradient Accumulation</mark>)</p></figcaption></figure>

## 🦥 **LoRA Hyperparameters in Unsloth**

The following demonstrates a standard configuration. **While Unsloth provides optimized defaults**, understanding these parameters is key to manual tuning.

<div data-full-width="false"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-9843f8cc26aac6445236250f5c32394186eace59%2Fnotebook_parameter_screenshott.png?alt=media" alt=""><figcaption></figcaption></figure></div>

1. ```python
   r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
   ```

   The rank (`r`) of the fine-tuning process. A larger rank uses more memory and will be slower, but can increase accuracy on complex tasks. We suggest ranks like 8 or 16 (for fast fine-tunes) and up to 128. Using a rank that is too large can cause overfitting and harm your model's quality.\\
2. ```python
   target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj",],
   ```

   For optimal performance, <mark style="background-color:blue;">**LoRA should be applied to all major linear layers**</mark>. [Research has shown](#lora-target-modules-and-qlora-vs-lora) that targeting all major layers is crucial for matching the performance of full fine-tuning. While it's possible to remove modules to reduce memory usage, we strongly advise against it to preserve maximum quality as the savings are minimal.\\
3. ```python
   lora_alpha = 16,
   ```

   A scaling factor that controls the strength of the fine-tuned adjustments. Setting it equal to the rank (`r`) is a reliable baseline. A popular and effective heuristic is to set it to double the rank (`r * 2`), which makes the model learn more aggressively by giving more weight to the LoRA updates. [More details here](#lora-alpha-and-rank-relationship).\\
4. ```python
   lora_dropout = 0, # Supports any, but = 0 is optimized
   ```

   A regularization technique that helps [prevent overfitting](#overfitting-poor-generalization-too-specialized) by randomly setting a fraction of the LoRA activations to zero during each training step. [Recent research suggests](https://arxiv.org/abs/2410.09692) that for **the short training runs** common in fine-tuning, `lora_dropout` may be an unreliable regularizer.\
   🦥 *Unsloth's internal code can optimize training when* `lora_dropout = 0`*, making it slightly faster, but we recommend a non-zero value if you suspect overfitting.*\\
5. ```python
   bias = "none",    # Supports any, but = "none" is optimized
   ```

   Leave this as `"none"` for faster training and reduced memory usage. This setting avoids training the bias terms in the linear layers, which adds trainable parameters for little to no practical gain.\\
6. ```python
   use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
   ```

   Options are `True`, `False`, and `"unsloth"`.\
   🦥 *We recommend* `"unsloth"` *as it reduces memory usage by an extra 30% and supports extremely long context fine-tunes. You can read more on* [*our blog post about long context training*](https://unsloth.ai/blog/long-context)*.*\\
7. ```python
   random_state = 3407,
   ```

   The seed to ensure deterministic, reproducible runs. Training involves random numbers, so setting a fixed seed is essential for consistent experiments.\\
8. ```python
   use_rslora = False,  # We support rank stabilized LoRA
   ```

   An advanced feature that implements [**Rank-Stabilized LoRA**](https://arxiv.org/abs/2312.03732). If set to `True`, the effective scaling becomes `lora_alpha / sqrt(r)` instead of the standard `lora_alpha / r`. This can sometimes improve stability, particularly for higher ranks. [More details here](#lora-alpha-and-rank-relationship).\\
9. ```python
   loftq_config = None, # And LoftQ
   ```

   An advanced technique, as proposed in [**LoftQ**](https://arxiv.org/abs/2310.08659), initializes LoRA matrices with the top 'r' singular vectors from the pretrained weights. This can improve accuracy but may cause a significant memory spike at the start of training.

### **Verifying LoRA Weight Updates:**

When validating that **LoRA** adapter weights have been updated after fine-tuning, avoid using **np.allclose()** for comparison. This method can miss subtle but meaningful changes, particularly in **LoRA A**, which is initialized with small Gaussian values. These changes may not register as significant under loose numerical tolerances. Thanks to [contributors](https://github.com/unslothai/unsloth/issues/3035) for this section.

To reliably confirm weight updates, we recommend:

* Using **checksum or hash comparisons** (e.g., MD5)
* Computing the **sum of absolute differences** between tensors
* Inspecting t**ensor statistics** (e.g., mean, variance) manually
* Or using **np.array\_equal()** if exact equality is expected

## :triangular\_ruler:LoRA Alpha and Rank relationship

{% hint style="success" %}
It's best to set `lora_alpha = 2 * lora_rank` or `lora_alpha = lora_rank`
{% endhint %}

{% columns %}
{% column width="50%" %}
$$
\hat{W} = W + \frac{\alpha}{\text{rank}} \times AB
$$

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-8e4f60c002f22e8ca9c534b48323e9e77e4b5ea6%2Fimage.png?alt=media" alt=""><figcaption><p>rsLoRA other scaling options. sqrt(r) is the best.</p></figcaption></figure>

$$
\hat{W}\_{\text{rslora}} = W + \frac{\alpha}{\sqrt{\text{rank}}} \times AB
$$
{% endcolumn %}

{% column %}
The formula for LoRA is on the left. We need to scale the thin matrices A and B by alpha divided by the rank. <mark style="background-color:blue;">**This means we should keep alpha/rank at least = 1**</mark>.

According to the [rsLoRA (rank stabilized lora) paper](https://arxiv.org/abs/2312.03732), we should instead scale alpha by the sqrt of the rank. Other options exist, but theoretically this is the optimum. The left plot shows other ranks and their perplexities (lower is better). To enable this, set `use_rslora = True` in Unsloth.

Our recommendation is to set the <mark style="background-color:green;">**alpha to equal to the rank, or at least 2 times the rank.**</mark> This means alpha/rank = 1 or 2.
{% endcolumn %}
{% endcolumns %}

## :dart: LoRA Target Modules and QLoRA vs LoRA

{% hint style="success" %}
Use:\
`target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",]` to target both **MLP** and **attention** layers to increase accuracy.

**QLoRA uses 4-bit precision**, reducing VRAM usage by over 75%.

**LoRA (16-bit)** is slightly more accurate and faster.
{% endhint %}

According to empirical experiments and research papers like the original [QLoRA paper](https://arxiv.org/pdf/2305.14314), it's best to apply LoRA to both attention and MLP layers.

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-16bef8165ccace21d0533f1941b8268a165c6a37%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
The chart shows RougeL scores (higher is better) for different target module configurations, comparing LoRA vs QLoRA.

The first 3 dots show:

1. **QLoRA-All:** LoRA applied to all FFN/MLP and Attention layers.\
   :fire: *This performs best overall.*
2. **QLoRA-FFN**: LoRA only on FFN.\
   Equivalent to: `gate_proj`, `up_proj`, `down_proj.`
3. **QLoRA-Attention**: LoRA applied only to Attention layers.\
   Equivalent to: `q_proj`, `k_proj`, `v_proj`, `o_proj`.
   {% endcolumn %}
   {% endcolumns %}

## :sunglasses: Training on completions only, masking out inputs

The [QLoRA paper](https://arxiv.org/pdf/2305.14314) shows that masking out inputs and **training only on completions** (outputs or assistant messages) can further **increase accuracy** by a few percentage points (*1%*). Below demonstrates how this is done in Unsloth:

{% columns %}
{% column %}
**NOT** training on completions only:

**USER:** <mark style="background-color:green;">Hello what is 2+2?</mark>\
**ASSISTANT:** <mark style="background-color:green;">The answer is 4.</mark>\
**USER:** <mark style="background-color:green;">Hello what is 3+3?</mark>\
**ASSISTANT:** <mark style="background-color:green;">The answer is 6.</mark>
{% endcolumn %}

{% column %}
**Training** on completions only:

**USER:** ~~Hello what is 2+2?~~\
**ASSISTANT:** <mark style="background-color:green;">The answer is 4.</mark>\
**USER:** ~~Hello what is 3+3?~~\
**ASSISTANT:** <mark style="background-color:green;">The answer is 6</mark><mark style="background-color:green;">**.**</mark>
{% endcolumn %}
{% endcolumns %}

The QLoRA paper states that **training on completions only** increases accuracy by quite a bit, especially for multi-turn conversational finetunes! We do this in our [conversational notebooks here](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(1B_and_3B\)-Conversational.ipynb).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-7e73b480d1db1dd3d52dd0d4a7e24caff6a54be0%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

To enable **training on completions** in Unsloth, you will need to define the instruction and assistant parts. :sloth: *We plan to further automate this for you in the future!*

For Llama 3, 3.1, 3.2, 3.3 and 4 models, you define the parts as follows:

```python
from unsloth.chat_templates import train_on_responses_only
trainer = train_on_responses_only(
    trainer,
    instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n",
    response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n",
)
```

For Gemma 2, 3, 3n models, you define the parts as follows:

```python
from unsloth.chat_templates import train_on_responses_only
trainer = train_on_responses_only(
    trainer,
    instruction_part = "<start_of_turn>user\n",
    response_part = "<start_of_turn>model\n",
)
```

## &#x20;:mag\_right:Training on assistant responses only for vision models, VLMs

For language models, we can use `from unsloth.chat_templates import train_on_responses_only` as described previously. For vision models, use the extra arguments as part of `UnslothVisionDataCollator` just like before!

{% code overflow="wrap" %}

```python
class UnslothVisionDataCollator:
def __init__(
    self,
    ...
    # from unsloth.chat_templates import train_on_responses_only
    # trainer = train_on_responses_only(
    #     trainer,
    #     instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n",
    #     response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n",
    # )
    train_on_responses_only = False, # EQUIVALENT to train_on_responses_only for LLMs
    instruction_part = None, # EQUIVALENT to train_on_responses_only(instruction_part = ...)
    response_part    = None, # EQUIVALENT to train_on_responses_only(response_part = ...)
    force_match      = True, # Match newlines as well!
)
```

{% endcode %}

For example for Llama 3.2 Vision:

```python
UnslothVisionDataCollator(
    model, tokenizer,
    ...
    train_on_responses_only = True,
    instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n",
    response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n",
    ...
)
```

## :key: **Avoiding Overfitting & Underfitting**

### **Overfitting** (Poor Generalization/Too Specialized)

The model memorizes the training data, including its statistical noise, and consequently fails to generalize to unseen data.

{% hint style="success" %}
If your training loss drops below 0.2, your model is likely **overfitting** — meaning it may perform poorly on unseen tasks.

One simple trick is LoRA alpha scaling — just multiply the alpha value of each LoRA matrix by 0.5. This effectively scales down the impact of fine-tuning.

**This is closely related to merging / averaging weights.**\
You can take the original base (or instruct) model, add the LoRA weights, then divide the result by 2. This gives you an averaged model — which is functionally equivalent to reducing the `alpha` by half.
{% endhint %}

**Solution:**

* **Adjust the learning rate:** A high learning rate often leads to overfitting, especially during short training runs. For longer training, a higher learning rate may work better. It’s best to experiment with both to see which performs best.
* **Reduce the number of training epochs**. Stop training after 1, 2, or 3 epochs.
* **Increase** `weight_decay`. A value of `0.01` or `0.1` is a good starting point.
* **Increase** `lora_dropout`. Use a value like `0.1` to add regularization.
* **Increase batch size or gradient accumulation steps**.
* **Dataset expansion** - make your dataset larger by combining or concatenating open source datasets with your dataset. Choose higher quality ones.
* **Evaluation early stopping** - enable evaluation and stop when the evaluation loss increases for a few steps.
* **LoRA Alpha Scaling** - scale the alpha down after training and during inference - this will make the finetune less pronounced.
* **Weight averaging** - literally add the original instruct model and the finetune and divide the weights by 2.

### **Underfitting** (Too Generic)

The model fails to capture the underlying patterns in the training data, often due to insufficient complexity or training duration.

**Solution:**

* **Adjust the Learning Rate:** If the current rate is too low, increasing it may speed up convergence, especially for short training runs. For longer runs, try lowering the learning rate instead. Test both approaches to see which works best.
* **Increase Training Epochs:** Train for more epochs, but monitor validation loss to avoid overfitting.
* **Increase LoRA Rank** (`r`) and alpha: Rank should at least equal to the alpha number, and rank should be bigger for smaller models/more complex datasets; it usually is between 4 and 64.
* **Use a More Domain-Relevant Dataset**: Ensure the training data is high-quality and directly relevant to the target task.
* **Decrease batch size to 1**. This will cause the model to update more vigorously.

{% hint style="success" %}
Fine-tuning has no single "best" approach, only best practices. Experimentation is key to finding what works for your specific needs. Our notebooks automatically set optimal parameters based on many papers research and our experiments, giving you a great starting point. Happy fine-tuning!
{% endhint %}

***Acknowledgements:** A huge thank you to* [*Eyera*](https://huggingface.co/Orenguteng) *for contributing to this guide!*


# What Model Should I Use for Fine-tuning?

## Llama, Qwen, Mistral, Phi or?

When preparing for fine-tuning, one of the first decisions you'll face is selecting the right model. Here's a step-by-step guide to help you choose:

{% stepper %}
{% step %}
**Choose a model that aligns with your usecase**

* E.g. For image-based training, select a vision model such as *Llama 3.2 Vision*. For code datasets, opt for a specialized model like *Qwen Coder 2.5*.
* **Licensing and Requirements**: Different models may have specific licensing terms and [system requirements](/docs/get-started/fine-tuning-for-beginners/unsloth-requirements#system-requirements). Be sure to review these carefully to avoid compatibility issues.
  {% endstep %}

{% step %}
**Assess your storage, compute capacity and dataset**

* Use our [VRAM guideline](/docs/get-started/fine-tuning-for-beginners/unsloth-requirements#approximate-vram-requirements-based-on-model-parameters) to determine the VRAM requirements for the model you’re considering.
* Your dataset will reflect the type of model you will use and amount of time it will take to train
  {% endstep %}

{% step %}
**Select a Model and Parameters**

* We recommend using the latest model for the best performance and capabilities. For instance, as of January 2025, the leading 70B model is *Llama 3.3*.
* You can stay up to date by exploring our [model catalog](/docs/get-started/unsloth-model-catalog) to find the newest and relevant options.
  {% endstep %}

{% step %}
**Choose Between Base and Instruct Models**

Further details below:
{% endstep %}
{% endstepper %}

## Instruct or Base Model?

When preparing for fine-tuning, one of the first decisions you'll face is whether to use an instruct model or a base model.

### Instruct Models

Instruct models are pre-trained with built-in instructions, making them ready to use without any fine-tuning. These models, including GGUFs and others commonly available, are optimized for direct usage and respond effectively to prompts right out of the box. Instruct models work with conversational chat templates like ChatML or ShareGPT.

### **Base Models**

Base models, on the other hand, are the original pre-trained versions without instruction fine-tuning. These are specifically designed for customization through fine-tuning, allowing you to adapt them to your unique needs. Base models are compatible with instruction-style templates like [Alpaca or Vicuna](/docs/basics/chat-templates), but they generally do not support conversational chat templates out of the box.

### Should I Choose Instruct or Base?

The decision often depends on the quantity, quality, and type of your data:

* **1,000+ Rows of Data**: If you have a large dataset with over 1,000 rows, it's generally best to fine-tune the base model.
* **300–1,000 Rows of High-Quality Data**: With a medium-sized, high-quality dataset, fine-tuning the base or instruct model are both viable options.
* **Less than 300 Rows**: For smaller datasets, the instruct model is typically the better choice. Fine-tuning the instruct model enables it to align with specific needs while preserving its built-in instructional capabilities. This ensures it can follow general instructions without additional input unless you intend to significantly alter its functionality.
* For information how how big your dataset should be, [see here](/docs/get-started/fine-tuning-llms-guide/datasets-guide#how-big-should-my-dataset-be)

## Fine-tuning models with Unsloth

You can change the model name to whichever model you like by matching it with model's name on Hugging Face e.g. 'unsloth/llama-3.1-8b-unsloth-bnb-4bit'.

We recommend starting with **Instruct models**, as they allow direct fine-tuning using conversational chat templates (ChatML, ShareGPT etc.) and require less data compared to **Base models** (which uses Alpaca, Vicuna etc). Learn more about the differences between [instruct and base models here](#instruct-or-base-model).

* Model names ending in **`unsloth-bnb-4bit`** indicate they are [**Unsloth dynamic 4-bit**](https://unsloth.ai/blog/dynamic-4bit) **quants**. These models consume slightly more VRAM than standard BitsAndBytes 4-bit models but offer significantly higher accuracy.
* If a model name ends with just **`bnb-4bit`**, without "unsloth", it refers to a standard BitsAndBytes 4-bit quantization.
* Models with **no suffix** are in their original **16-bit or 8-bit formats**. While they are the original models from the official model creators, we sometimes include important fixes - such as chat template or tokenizer fixes. So it's recommended to use our versions when available.

### Experimentation is Key

{% hint style="info" %}
We recommend experimenting with both models when possible. Fine-tune each one and evaluate the outputs to see which aligns better with your goals.
{% endhint %}


# Tutorial: How to Finetune Llama-3 and Use In Ollama

Beginner's Guide for creating a customized personal assistant (like ChatGPT) to run locally on Ollama

By the end of this tutorial, you will create a custom chatbot by **finetuning Llama-3** with [**Unsloth**](https://github.com/unslothai/unsloth) for free. It can run locally via [**Ollama**](https://github.com/ollama/ollama) on your PC, or in a free GPU instance through [**Google Colab**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Ollama.ipynb). You will be able to interact with the chatbot interactively like below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-cf9aed2029e54afbb65889b480134e6d5e1cf3a7%2FAssistant%20example.png?alt=media" alt=""><figcaption></figcaption></figure>

**Unsloth** makes finetuning much easier, and can automatically export the finetuned model to **Ollama** with integrated automatic `Modelfile` creation! If you need help, you can join our Discord server: <https://discord.com/invite/unsloth>

{% hint style="warning" %}
**If you’d like to copy or save the code, everything is available in our** [**Ollama Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Ollama.ipynb)**. You can use it directly there or adapt it for your local setup:** [**https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3\_(8B)-Ollama.ipynb**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Ollama.ipynb)
{% endhint %}

## 1. What is Unsloth?

[Unsloth](https://github.com/unslothai/unsloth) makes finetuning LLMs like Llama-3, Mistral, Phi-3 and Gemma 2x faster, use 70% less memory, and with no degradation in accuracy! We will be using Google Colab which provides a free GPU during this tutorial. You can access our free notebooks below:

* [Ollama Llama-3 Alpaca](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Ollama.ipynb) (notebook which we will be using)
* [CSV/Excel Ollama Guide](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing)

#### ***You will also need to login into your Google account!***

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-bca149bda83c2192982b136cfeb096999c469a2e%2FColab%20Screen.png?alt=media" alt=""><figcaption></figcaption></figure>

## 2. What is Ollama?

[Ollama ](https://github.com/ollama/ollama)allows you to run language models from your own computer in a quick and simple way! It quietly launches a program which can run a language model like Llama-3 in the background. If you suddenly want to ask the language model a question, you can simply submit a request to Ollama, and it'll quickly return the results to you! We'll be using Ollama as our inference engine!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-fd25844766001d93ed0949fc8f57957f49b1e6e5%2FOllama.png?alt=media" alt=""><figcaption></figcaption></figure>

## 3. Install Unsloth

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4d1b1778f3c8bde62a40130d7b4395b8bb1ce90f%2FColab%20Options.png?alt=media" alt=""><figcaption></figcaption></figure>

If you have never used a Colab notebook, a quick primer on the notebook itself:

1. **Play Button at each "cell".** Click on this to run that cell's code. You must not skip any cells and you must run every cell in chronological order. If you encounter any errors, simply rerun the cell you did not run before. Another option is to click CTRL + ENTER if you don't want to click the play button.
2. **Runtime Button in the top toolbar.** You can also use this button and hit "Run all" to run the entire notebook in 1 go. This will skip all the customization steps, and can be a good first try.
3. **Connect / Reconnect T4 button.** You can click here for more advanced system statistics.

The first installation cell looks like below: Remember to click the PLAY button in the brackets \[ ]. We grab our open source Github package, and install some other packages.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-3ae88d2cf9ba1c59b13d701864750ac311a60426%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## 4. Selecting a model to finetune

Let's now select a model for finetuning! We defaulted to Llama-3 from Meta / Facebook which was trained on a whopping 15 trillion "tokens". Assume a token is like 1 English word. That's approximately 350,000 thick Encyclopedias worth! Other popular models include Mistral, Phi-3 (trained using GPT-4 output) and Gemma from Google (13 trillion tokens!).

Unsloth supports these models and more! In fact, simply type a model from the Hugging Face model hub to see if it works! We'll error out if it doesn't work.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4fb10a1ce3e457310c11f74ca5b6347ad556fab0%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

There are 3 other settings which you can toggle:

1. ```python
   max_seq_length = 2048
   ```

   This determines the context length of the model. Gemini for example has over 1 million context length, whilst Llama-3 has 8192 context length. We allow you to select ANY number - but we recommend setting it 2048 for testing purposes. Unsloth also supports very long context finetuning, and we show we can provide 4x longer context lengths than the best.
2. ```python
   dtype = None
   ```

   Keep this as None, but you can select torch.float16 or torch.bfloat16 for newer GPUs.
3. ```python
   load_in_4bit = True
   ```

   We do finetuning in 4 bit quantization. This reduces memory usage by 4x, allowing us to actually do finetuning in a free 16GB memory GPU. 4 bit quantization essentially converts weights into a limited set of numbers to reduce memory usage. A drawback of this is there is a 1-2% accuracy degradation. Set this to False on larger GPUs like H100s if you want that tiny extra accuracy.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a44ac84348a2c5973dd542866c4c6727a00b3744%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

If you run the cell, you will get some print outs of the Unsloth version, which model you are using, how much memory your GPU has, and some other statistics. Ignore this for now.

## 5. Parameters for finetuning

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-495edc79c5353f0f47c1eea58df045631bfef1e0%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Now to customize your finetune, you can edit the numbers above, but you can ignore it, since we already select quite reasonable numbers.

The goal is to change these numbers to increase accuracy, but also **counteract over-fitting**. Over-fitting is when you make the language model memorize a dataset, and not be able to answer novel new questions. We want to a final model to answer unseen questions, and not do memorization.

1. ```python
   r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
   ```

   The rank of the finetuning process. A larger number uses more memory and will be slower, but can increase accuracy on harder tasks. We normally suggest numbers like 8 (for fast finetunes), and up to 128. Too large numbers can causing over-fitting, damaging your model's quality.
2. ```python
   target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj",],
   ```

   We select all modules to finetune. You can remove some to reduce memory usage and make training faster, but we highly do not suggest this. Just train on all modules!
3. ```python
   lora_alpha = 16,
   ```

   The scaling factor for finetuning. A larger number will make the finetune learn more about your dataset, but can promote over-fitting. We suggest this to equal to the rank `r`, or double it.
4. ```python
   lora_dropout = 0, # Supports any, but = 0 is optimized
   ```

   Leave this as 0 for faster training! Can reduce over-fitting, but not that much.
5. ```python
   bias = "none",    # Supports any, but = "none" is optimized
   ```

   Leave this as 0 for faster and less over-fit training!
6. ```python
   use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
   ```

   Options include `True`, `False` and `"unsloth"`. We suggest `"unsloth"` since we reduce memory usage by an extra 30% and support extremely long context finetunes.You can read up here: <https://unsloth.ai/blog/long-context> for more details.
7. ```python
   random_state = 3407,
   ```

   The number to determine deterministic runs. Training and finetuning needs random numbers, so setting this number makes experiments reproducible.
8. ```python
   use_rslora = False,  # We support rank stabilized LoRA
   ```

   Advanced feature to set the `lora_alpha = 16` automatically. You can use this if you want!
9. ```python
   loftq_config = None, # And LoftQ
   ```

   Advanced feature to initialize the LoRA matrices to the top r singular vectors of the weights. Can improve accuracy somewhat, but can make memory usage explode at the start.

## 6. Alpaca Dataset

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1d66d8714e44d90513dd87b9356eec67886ab3f7%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

We will now use the Alpaca Dataset created by calling GPT-4 itself. It is a list of 52,000 instructions and outputs which was very popular when Llama-1 was released, since it made finetuning a base LLM be competitive with ChatGPT itself.

You can access the GPT4 version of the Alpaca dataset here: <https://huggingface.co/datasets/vicgalle/alpaca-gpt4>. An older first version of the dataset is here: <https://github.com/tatsu-lab/stanford_alpaca>. Below shows some examples of the dataset:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-0dde50e386e7b245d3e8a57e10a4a81755b3769a%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

You can see there are 3 columns in each row - an instruction, and input and an output. We essentially combine each row into 1 large prompt like below. We then use this to finetune the language model, and this made it very similar to ChatGPT. We call this process **supervised instruction finetuning**.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-8b3663c5d80adcb935ff77661500f08e13c9af2d%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## 7. Multiple columns for finetuning

But a big issue is for ChatGPT style assistants, we only allow 1 instruction / 1 prompt, and not multiple columns / inputs. For example in ChatGPT, you can see we must submit 1 prompt, and not multiple prompts.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-d90162c2685ced871f4151369aadcaee40a9c54f%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

This essentially means we have to "merge" multiple columns into 1 large prompt for finetuning to actually function!

For example the very famous Titanic dataset has many many columns. Your job was to predict whether a passenger has survived or died based on their age, passenger class, fare price etc. We can't simply pass this into ChatGPT, but rather, we have to "merge" this information into 1 large prompt.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a2df04874bfc879182cb66c789341d49700227ea%2FMerge.png?alt=media" alt=""><figcaption></figcaption></figure>

For example, if we ask ChatGPT with our "merged" single prompt which includes all the information for that passenger, we can then ask it to guess or predict whether the passenger has died or survived.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-b3da2b36afe37469cd3962f37186e758871864a5%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Other finetuning libraries require you to manually prepare your dataset for finetuning, by merging all your columns into 1 prompt. In Unsloth, we simply provide the function called `to_sharegpt` which does this in 1 go!

To access the Titanic finetuning notebook or if you want to upload a CSV or Excel file, go here: <https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-62b94dc44f2e343020d31de575f52eb22be4b0fc%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Now this is a bit more complicated, since we allow a lot of customization, but there are a few points:

* You must enclose all columns in curly braces `{}`. These are the column names in the actual CSV / Excel file.
* Optional text components must be enclosed in `[[]]`. For example if the column "input" is empty, the merging function will not show the text and skip this. This is useful for datasets with missing values.
* Select the output or target / prediction column in `output_column_name`. For the Alpaca dataset, this will be `output`.

For example in the Titanic dataset, we can create a large merged prompt format like below, where each column / piece of text becomes optional.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e6228cf6e5c0bb4e4b45e6f3e045910d567c33d2%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

For example, pretend the dataset looks like this with a lot of missing data:

| Embarked | Age | Fare |
| -------- | --- | ---- |
| S        | 23  |      |
|          | 18  | 7.25 |

Then, we do not want the result to be:

1. The passenger embarked from S. Their age is 23. Their fare is **EMPTY**.
2. The passenger embarked from **EMPTY**. Their age is 18. Their fare is $7.25.

Instead by optionally enclosing columns using `[[]]`, we can exclude this information entirely.

1. \[\[The passenger embarked from S.]] \[\[Their age is 23.]] \[\[Their fare is **EMPTY**.]]
2. \[\[The passenger embarked from **EMPTY**.]] \[\[Their age is 18.]] \[\[Their fare is $7.25.]]

becomes:

1. The passenger embarked from S. Their age is 23.
2. Their age is 18. Their fare is $7.25.

## 8. Multi turn conversations

An issue if you didn't notice is the Alpaca dataset is single turn, whilst remember using ChatGPT was interactive and you can talk to it in multiple turns. For example, the left is what we want, but the right which is the Alpaca dataset only provides singular conversations. We want the finetuned language model to somehow learn how to do multi turn conversations just like ChatGPT.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2a65cd74ddd03a6bcbbc9827d9d034e4879a8e6a%2Fdiff.png?alt=media" alt=""><figcaption></figcaption></figure>

So we introduced the `conversation_extension` parameter, which essentially selects some random rows in your single turn dataset, and merges them into 1 conversation! For example, if you set it to 3, we randomly select 3 rows and merge them into 1! Setting them too long can make training slower, but could make your chatbot and final finetune much better!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2b1b3494b260f1102942d86143a885225c6a06f2%2Fcombine.png?alt=media" alt=""><figcaption></figcaption></figure>

Then set `output_column_name` to the prediction / output column. For the Alpaca dataset dataset, it would be the output column.

We then use the `standardize_sharegpt` function to just make the dataset in a correct format for finetuning! Always call this!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-7bf83bf802191bda9e417bbe45afa181e7f24f38%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## 9. Customizable Chat Templates

We can now specify the chat template for finetuning itself. The very famous Alpaca format is below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-59737e6dcb09fed15487d5a57c69f07cb40bb8e7%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

But remember we said this was a bad idea because ChatGPT style finetunes require only 1 prompt? Since we successfully merged all dataset columns into 1 using Unsloth, we essentially can create the below style chat template with 1 input column (instruction) and 1 output:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-d54582ae98c396d51bfb85628b46c54f2517d030%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

We just require you must put a `{INPUT}` field for the instruction and an `{OUTPUT}` field for the model's output field. We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT. For example, below are some cool examples which you can customize the chat template to be:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-cc455dc380d3d44ef136e485754964159dc773d8%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

For the ChatML format used in OpenAI models:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-15bfca9cfadf10d54b4d3f66e3050044317d62c5%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Or you can use the Llama-3 template itself (which only functions by using the instruct version of Llama-3): We in fact allow an optional `{SYSTEM}` field as well which is useful to customize a system prompt just like in ChatGPT.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-80a2ed4de2ca323ac192c513cac65e9e8bf475db%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Or in the Titanic prediction task where you had to predict if a passenger died or survived in this Colab notebook which includes CSV and Excel uploading: <https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-20911ab305c1a10e85859c703157b80175141eb1%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## 10. Train the model

Let's train the model now! We normally suggest people to not edit the below, unless if you want to finetune for longer steps or want to train on large batch sizes.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-f55503cea4d84b5885d0bcea0563fd716a0d2ed6%2Fimage%20(43).png?alt=media" alt=""><figcaption></figcaption></figure>

We do not normally suggest changing the parameters above, but to elaborate on some of them:

1. ```python
   per_device_train_batch_size = 2,
   ```

   Increase the batch size if you want to utilize the memory of your GPU more. Also increase this to make training more smooth and make the process not over-fit. We normally do not suggest this, since this might make training actually slower due to padding issues. We normally instead ask you to increase `gradient_accumulation_steps` which just does more passes over the dataset.
2. ```python
   gradient_accumulation_steps = 4,
   ```

   Equivalent to increasing the batch size above itself, but does not impact memory consumption! We normally suggest people increasing this if you want smoother training loss curves.
3. ```python
   max_steps = 60, # num_train_epochs = 1,
   ```

   We set steps to 60 for faster training. For full training runs which can take hours, instead comment out `max_steps`, and replace it with `num_train_epochs = 1`. Setting it to 1 means 1 full pass over your dataset. We normally suggest 1 to 3 passes, and no more, otherwise you will over-fit your finetune.
4. ```python
   learning_rate = 2e-4,
   ```

   Reduce the learning rate if you want to make the finetuning process slower, but also converge to a higher accuracy result most likely. We normally suggest 2e-4, 1e-4, 5e-5, 2e-5 as numbers to try.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-feb9b0f5763d41cecaec9a3a9cd227ad918f0ca7%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

You’ll see a log of numbers during training. This is the training loss, which shows how well the model is learning from your dataset. For many cases, a loss around 0.5 to 1.0 is a good sign, but it depends on your dataset and task. If the loss is not going down, you might need to adjust your settings. If the loss goes to 0, that could mean overfitting, so it's important to check validation too.

## 11. Inference / running the model

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-f2d5f23fa62ec89e06bf20fea433f9a1e42a2fe3%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Now let's run the model after we completed the training process! You can edit the yellow underlined part! In fact, because we created a multi turn chatbot, we can now also call the model as if it saw some conversations in the past like below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-cdf5d779635901dce7793df92531dbf3caf0fb0a%2Fimage%20(47).png?alt=media" alt=""><figcaption></figcaption></figure>

Reminder Unsloth itself provides **2x faster inference** natively as well, so always do not forget to call `FastLanguageModel.for_inference(model)`. If you want the model to output longer responses, set `max_new_tokens = 128` to some larger number like 256 or 1024. Notice you will have to wait longer for the result as well!

## 12. Saving the model

We can now save the finetuned model as a small 100MB file called a LoRA adapter like below. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a Hugging Face token via <https://huggingface.co/settings/tokens> and add your token!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-8c577103f7c4fe883cabaf35c8437307c6501686%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

After saving the model, we can again use Unsloth to run the model itself! Use `FastLanguageModel` again to call it for inference!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1a1be852ca551240bdce47cf99e6ccd7d31c1326%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## 13. Exporting to Ollama

Finally we can export our finetuned model to Ollama itself! First we have to install Ollama in the Colab notebook:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-24f9429ed4a8b3a630dc8f68dcf81555da0a80ee%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Then we export the finetuned model we have to llama.cpp's GGUF formats like below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-56991ea7e2685bb9905af9baf2f3f685123dcdd8%2Fimage%20(52).png?alt=media" alt=""><figcaption></figcaption></figure>

Reminder to convert `False` to `True` for 1 row, and not change every row to `True`, or else you'll be waiting for a very time! We normally suggest the first row getting set to `True`, so we can export the finetuned model quickly to `Q8_0` format (8 bit quantization). We also allow you to export to a whole list of quantization methods as well, with a popular one being `q4_k_m`.

Head over to <https://github.com/ggml-org/llama.cpp> to learn more about GGUF. We also have some manual instructions of how to export to GGUF if you want here: <https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf>

You will see a long list of text like below - please wait 5 to 10 minutes!!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-271b392fdafd0e7d01c525d7a11a97ee5c34b713%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

And finally at the very end, it'll look like below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a554bd388fd0394dd8cdef85fd9d208bfd7feee7%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

Then, we have to run Ollama itself in the background. We use `subprocess` because Colab doesn't like asynchronous calls, but normally one just runs `ollama serve` in the terminal / command prompt.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e431609dfc5c742f0b5ab2388dbbd0d8e15c7670%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## 14. Automatic `Modelfile` creation

The trick Unsloth provides is we automatically create a `Modelfile` which Ollama requires! This is a just a list of settings and includes the chat template which we used for the finetune process! You can also print the `Modelfile` generated like below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-6945ba10a2e25cfc198848c0e863001375c32c4c%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

We then ask Ollama to create a model which is Ollama compatible, by using the `Modelfile`

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-d431a64613b39d913d1780c22cde37edc6564272%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

## 15. Ollama Inference

And we can now call the model for inference if you want to do call the Ollama server itself which is running on your own local machine / in the free Colab notebook in the background. Remember you can edit the yellow underlined part.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-49b93efa192fdd741f3ac8484cef8c3fd7415283%2FInference.png?alt=media" alt=""><figcaption></figcaption></figure>

## 16. Interactive ChatGPT style

But to actually run the finetuned model like a ChatGPT, we have to do a bit more! First click the terminal icon![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-9c24108bc5152f946a7afab054974890318d2c02%2Fimage.png?alt=media) and a Terminal will pop up. It's on the left sidebar.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2239315eff2820bf9f224975f0b184d51bd89cb7%2FWhere_Terminal.png?alt=media" alt=""><figcaption></figcaption></figure>

Then, you might have to press ENTER twice to remove some weird output in the Terminal window. Wait a few seconds and type `ollama run unsloth_model` then hit ENTER.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e83ac484e4257eacad1c7d033811d2ece59a444c%2FTerminal_Type.png?alt=media" alt=""><figcaption></figcaption></figure>

And finally, you can interact with the finetuned model just like an actual ChatGPT! Hit CTRL + D to exit the system, and hit ENTER to converse with the chatbot!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-120703475091e1ce74a38a05949ae51af0a36f72%2FAssistant.png?alt=media" alt=""><figcaption></figcaption></figure>

## You've done it!

You've successfully finetuned a language model and exported it to Ollama with Unsloth 2x faster and with 70% less VRAM! And all this for free in a Google Colab notebook!

If you want to learn how to do reward modelling, do continued pretraining, export to vLLM or GGUF, do text completion, or learn more about finetuning tips and tricks, head over to our [Github](https://github.com/unslothai/unsloth#-finetune-for-free).

If you need any help on finetuning, you can also join our Discord server [here](https://discord.gg/unsloth). If you want help with Ollama, you can also join their server [here](https://discord.gg/ollama).

And finally, we want to thank you for reading and following this far! We hope this made you understand some of the nuts and bolts behind finetuning language models, and we hope this was useful!

To access our Alpaca dataset example click [here](https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing), and our CSV / Excel finetuning guide is [here](https://colab.research.google.com/drive/1VYkncZMfGFkeCEgN2IzbZIKEDkyQuJAS?usp=sharing).


# Reinforcement Learning (RL) Guide

Learn all about Reinforcement Learning (RL) and how to train your own DeepSeek-R1 reasoning model with Unsloth using GRPO. A complete guide from beginner to advanced.

Reinforcement Learning is where an "agent" learns to make decisions by interacting with an environment and receiving **feedback** in the form of **rewards** or **penalties**.

* **Action:** What the model generates (e.g. a sentence).
* **Reward:** A signal indicating how good or bad the model's action was (e.g. did the response follow instructions? was it helpful?).
* **Environment:** The scenario or task the model is working on (e.g. answering a user’s question).

### :sloth:What you will learn

1. What is RL? RLVR? PPO? GRPO? RLHF? RFT? Is <mark style="background-color:green;">**"Luck is All You Need?"**</mark> for RL?
2. What is an environment? Agent? Action? Reward function? Rewards?

This article covers everything (from beginner to advanced) you need to know about GRPO, Reinforcement Learning (RL) and reward functions, along with tips, and the basics of using GRPO with [Unsloth](https://github.com/unslothai/unsloth). If you're looking for a step-by-step tutorial for using GRPO, see our guide [here](/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo).

{% hint style="success" %}
**Jan 15, 2026 update:** [Ultra long context RL](/docs/get-started/reinforcement-learning-rl-guide/grpo-long-context) is here! Train gpt-oss with a 380K context window.

**Nov 26, 2025 update:** We're introducing FP8 precision RL and GRPO in Unsloth! [Read blog](/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning)
{% endhint %}

## :question:What is Reinforcement Learning (RL)?

The goal of RL is to:

1. **Increase the chance of seeing&#x20;**<mark style="background-color:green;">**"good"**</mark>**&#x20;outcomes.**
2. **Decrease the chance of seeing&#x20;**<mark style="background-color:red;">**"bad"**</mark>**&#x20;outcomes.**

**That's it!** There are intricacies on what "good" and "bad" means, or how do we go about "increasing" or "decreasing" it, or what even "outcomes" means.

{% columns %}
{% column width="50%" %}
For example, in the **Pacman game**:

1. The <mark style="background-color:green;">**environment**</mark> is the game world.
2. The <mark style="background-color:blue;">**actions**</mark> you can take are UP, LEFT, RIGHT and DOWN.
3. The <mark style="background-color:purple;">**rewards**</mark> are good if you eat a cookie, or bad if you hit one of the squiggly enemies.
4. In RL, you can't know the "best action" you can take, but you can observe intermediate steps, or the final game state (win or lose)
   {% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e853f7e6da505ee587642314b98180ebf840252c%2FRL%20Game.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-30bade1550c877bb7f79075c80ac79476b0ecd76%2FMath%20RL.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
Another example is imagine you are given the question: <mark style="background-color:blue;">**"What is 2 + 2?"**</mark> (4) An unaligned language model will spit out 3, 4, C, D, -10, literally anything.

1. Numbers are better than C or D right?
2. Getting 3 is better than say 8 right?
3. Getting 4 is definitely correct.

We just designed a <mark style="background-color:orange;">**reward function**</mark>!
{% endcolumn %}
{% endcolumns %}

### :person\_running:From RLHF, PPO to GRPO and RLVR

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-5d0c90e4b45507d3e12c8b938cbd1679cd38f4f9%2FRLHF.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
OpenAI popularized the concept of [RLHF](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback) (Reinforcement Learning from Human Feedback), where we train an <mark style="background-color:red;">**"agent"**</mark> to produce outputs to a question (the <mark style="background-color:yellow;">**state**</mark>) that are rated more useful by human beings.

The thumbs up and down in ChatGPT for example can be used in the RLHF process.
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1e1dff9c921e787e669dee79c41a76db89e882e7%2FPPO.png?alt=media" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-f6156f2c519baf81e6ef286476f4092037303799%2FPPO%20formula.png?alt=media" alt=""><figcaption><p>PPO formula</p></figcaption></figure>

The clip(..., 1-e, 1+e) term is used to force PPO not to take too large changes. There is also a KL term with beta set to > 0 to force the model not to deviate too much away.
{% endcolumn %}

{% column %}
In order to do RLHF, [<mark style="background-color:red;">**PPO**</mark>](https://en.wikipedia.org/wiki/Proximal_policy_optimization) (Proximal policy optimization) was developed. The <mark style="background-color:blue;">**agent**</mark> is the language model in this case. In fact it's composed of 3 systems:

1. The **Generating Policy (current trained model)**
2. The **Reference Policy (original model)**
3. The **Value Model (average reward estimator)**

We use the **Reward Model** to calculate the reward for the current environment, and our goal is to **maximize this**!

The formula for PPO looks quite complicated because it was designed to be stable. Visit our [AI Engineer talk](https://docs.unsloth.ai/ai-engineers-2025) we gave in 2025 about RL for more in depth maths derivations about PPO.
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4f4e188edbcad4f53aaa4a626bc5b2fd01334574%2FGRPO%20%2B%20RLVR.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
DeepSeek developed [<mark style="background-color:red;">**GRPO**</mark>](https://unsloth.ai/blog/grpo) (Group Relative Policy Optimization) to train their R1 reasoning models. The key differences to PPO are:

1. The **Value Model is removed,** replaced with statistics from calling the reward model multiple times.
2. The **Reward Model is removed** and replaced with just custom reward function which <mark style="background-color:blue;">**RLVR**</mark> can be used.
   {% endcolumn %}
   {% endcolumns %}

This means GRPO is extremely efficient. Previously PPO needed to train multiple models - now with the reward model and value model removed, we can save memory and speed up everything.

<mark style="background-color:orange;">**RLVR (Reinforcement Learning with Verifiable Rewards)**</mark> allows us to reward the model based on tasks with easy to verify solutions. For example:

1. Maths equations can be easily verified. Eg 2+2 = 4.
2. Code output can be verified as having executed correctly or not.
3. Designing verifiable reward functions can be tough, and so most examples are math or code.
4. Use-cases for GRPO isn’t just for code or math—its reasoning process can enhance tasks like email automation, database retrieval, law, and medicine, greatly improving accuracy based on your dataset and reward function - the trick is to define a <mark style="background-color:yellow;">**rubric - ie a list of smaller verifiable rewards, and not a final all consuming singular reward.**</mark> OpenAI popularized this in their [reinforcement learning finetuning (RFT)](https://platform.openai.com/docs/guides/reinforcement-fine-tuning) offering for example.

{% columns %}
{% column %} <mark style="background-color:red;">**Why "Group Relative"?**</mark>

GRPO removes the value model entirely, but we still need to estimate the <mark style="background-color:yellow;">**"average reward"**</mark> given the current state.

The **trick is to sample the LLM**! We then calculate the average reward through statistics of the sampling process across multiple different questions.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-29e188e5adc6de1e62c841e6cd9e34a2dae4994a%2FGroup%20Relative.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}
For example for "What is 2+2?" we sample 4 times. We might get 4, 3, D, C. We then calculate the reward for each of these answers, then calculate the **average reward** and **standard deviation**, then <mark style="background-color:red;">**Z-score standardize**</mark> this!

This creates the <mark style="background-color:blue;">**advantages A**</mark>, which we will use in replacement of the value model. This saves a lot of memory!
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-d40a73cd48b05b9205810a1946f4fc1dce81ae7d%2FStatistics.png?alt=media" alt=""><figcaption><p>GRPO advantage calculation</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### :fingers\_crossed:Luck (well Patience) Is All You Need

The trick of RL is you need 2 things only:

1. A question or instruction eg "What is 2+2?" "Create a Flappy Bird game in Python"
2. A reward function and verifier to verify if the output is good or bad.

With only these 2, we can essentially **call a language model an infinite times** until we get a good answer. For example for "What is 2+2?", an untrained bad language model will output:

***0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31\*\*\*\*\*\*\*\*&#x20;**<mark style="color:green;">**then suddenly 4**</mark>**.***

***The reward signal was 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0\*\*\*\*\*\*\*\*&#x20;**<mark style="color:green;">**then suddenly 1.**</mark>*

So by luck and by chance, RL managed to find the correct answer across multiple <mark style="background-color:yellow;">**rollouts**</mark>. Our goal is we want to see the good answer 4 more, and the rest (the bad answers) much less.

<mark style="color:blue;">**So the goal of RL is to be patient - in the limit, if the probability of the correct answer is at least a small number (not zero), it's just a waiting game - you will 100% for sure encounter the correct answer in the limit.**</mark>

<mark style="background-color:blue;">**So I like to call it as "Luck Is All You Need" for RL.**</mark>

<mark style="background-color:orange;">**Well a better phrase is "Patience is All You Need" for RL.**</mark>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4f0cb4803aa22583e88dfa8de8061b66bbe6a6b1%2FLuck%20is%20all%20you%20need.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

RL essentially provides us a trick - instead of simply waiting for infinity, we do get "bad signals" ie bad answers, and we can essentially "guide" the model to already try not generating bad solutions. This means although you waited very long for a "good" answer to pop up, the model already has been changed to try its best not to output bad answers.

In the "What is 2+2?" example - ***0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31\*\*\*\*\*\*\*\*&#x20;**<mark style="color:green;">**then suddenly 4**</mark>**.***

Since we got bad answers, RL will influence the model to try NOT to output bad answers. This means over time, we are carefully "pruning" or moving the model's output distribution away from bad answers. This means RL is <mark style="color:blue;">**efficient**</mark>, since we are NOT just waiting for infinity, but we are actively trying to "push" the model to go as much as possible to the "correct answer space".

{% hint style="danger" %}
**If the probability is always 0, then RL will never work**. This is also why people like to do RL from an already instruction finetuned model, which can partially follow instructions reasonably well - this boosts the probability most likely above 0.
{% endhint %}

## :sloth:What Unsloth offers for RL

* With 15GB VRAM, Unsloth allows you to transform any model up to 17B parameters like Llama 3.1 (8B), Phi-4 (14B), Mistral (7B) or Qwen2.5 (7B) into a reasoning model
* **Unsloth now supports** [**RL for Vision/multimodal**](/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl) **models!**
* **Minimum requirement:** Just  5GB VRAM is enough to train your own reasoning model locally (for any model with 1.5B parameters or less)

{% columns %}
{% column %}
{% content-ref url="/pages/QbifW3DsTPYvRJcVkXYz" %}
[Tutorial: GRPO Training](/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo)
{% endcontent-ref %}
{% endcolumn %}

{% column %}
{% content-ref url="/pages/aV6S9cmDmSv5ky4ZCo5d" %}
[Vision RL](/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

{% hint style="info" %}
For **advanced GRPO** documentation on batching, generation and training parameters, [read our guide!](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation)
{% endhint %}

### GRPO notebooks:

| [**Gemma 4 E2B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)_Reinforcement_Learning_Sudoku_Game.ipynb) - **new** | [Qwen3-VL-8B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb) - Vision GSPO | [Gemma 3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision-GRPO.ipynb) - Vision GSPO         |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| [**Qwen3.5 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)_Vision_GRPO.ipynb) **- Vision -&#x20;*****new***   | [gpt-oss-20b](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) GSPO                 | [Llama 3.2 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_\(3B\)_GRPO_LoRA.ipynb) - Advanced |
| [Qwen3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-GRPO.ipynb) - Advanced                                     | [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_\(14B\)-GRPO.ipynb)                        | [DeepSeek-R1-0528-Qwen3-8B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_\(8B\)_GRPO.ipynb) |
| [Mistral v0.3 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-GRPO.ipynb)                                  | [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_\(8B\)-GRPO.ipynb)                   | [Qwen3-8B - **FP8**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) (L4)                    |

{% hint style="success" %}
We support [**GSPO**](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation/gspo-reinforcement-learning) and most other new GRPO techniques. You can play with the following arguments in GRPOConfig to enable:

```python
epsilon=0.2,
epsilon_high=0.28, # one sided
delta=1.5 # two sided

loss_type='gspo',
# or:
loss_type='grpo',
# or:
loss_type='dr_grpo',

mask_truncated_completions=True,
```

{% endhint %}

* If you're not getting any reasoning, make sure you have enough training steps and ensure your [reward function/verifier](#reward-functions-verifier) is working. We provide examples for reward functions [here](#reward-function-examples).
* Previous demonstrations show that you could achieve your own "aha" moment with Qwen2.5 (3B) - but it required 2xA100 GPUs (160GB VRAM). Now, with Unsloth, you can achieve the same "aha" moment using just a single 5GB VRAM GPU.
* Previously, GRPO was only supported for full fine-tuning, but we've made it work with QLoRA and LoRA
* On [**20K context lengths**](#grpo-requirement-guidelines) for example with 8 generations per prompt, Unsloth uses only 54.3GB of VRAM for Llama 3.1 (8B), whilst standard implementations (+ Flash Attention 2) take **510.8GB (90% less for Unsloth)**.
* Please note, this isn’t fine-tuning DeepSeek’s R1 distilled models or using distilled data from R1 for tuning which Unsloth already supported. This is converting a standard model into a full-fledged reasoning model using GRPO.

In a test example, even though we only trained Phi-4 with 100 steps using GRPO, the results are already clear. The model without GRPO does not have the thinking token, whilst the one trained with GRPO does and also has the correct answer.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-5ae836156344a7c22241d0f76dbea09d58e04f8f%2Fprompt%20only%20example.png?alt=media" alt=""><figcaption></figcaption></figure>

## :computer:Training with GRPO

For a tutorial on how to transform any open LLM into a reasoning model using Unsloth & GRPO, [see here](/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo).

{% hint style="success" %}
For **advanced GRPO** documentation on batching, generation and training parameters, [read our guide!](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation)
{% endhint %}

### **How GRPO Trains a Model**

1. For each question-answer pair, the model generates multiple possible responses (e.g., 8 variations).
2. Each response is evaluated using reward functions.
3. Training Steps:
   * If you have 300 rows of data, that's 300 training steps (or 900 steps if trained for 3 epochs).
   * You can increase the number of generated responses per question (e.g., from 8 to 16).
4. The model learns by updating its weights every step.

{% hint style="warning" %}
If you're having issues with your GRPO model not learning, we'd highly recommend to use our [Advanced GRPO notebooks](/docs/get-started/unsloth-notebooks#grpo-reasoning-notebooks) as it has a much better reward function and you should see results much faster and frequently.
{% endhint %}

### Basics/Tips

* Wait for at least **300 steps** for the reward to actually increase. In order to get decent results, you may need to trade for a minimum of 12 hours (this is how GRPO works), but keep in mind this isn't compulsory as you can stop at anytime.
* For optimal results have at least **500 rows of data**. You can try with even 10 rows of data but it's better to have more.
* Each training run will always be different depending on your model, data, reward function/verifier etc. so though 300 steps is what we wrote as the minimum, sometimes it might be 1000 steps or more. So, it depends on various factors.
* If you're using GRPO with Unsloth locally, please "pip install diffusers" as well if you get an error. Please also use the latest version of vLLM.
* It’s advised to apply GRPO to a model at least **1.5B in parameters** to correctly generate thinking tokens as smaller models may not.
* For GRPO's [**GPU VRAM requirements**](#grpo-requirement-guidelines) **for QLoRA 4-bit**, the general rule is the model parameters = the amount of VRAM you will need (you can use less VRAM but this just to be safe). The more context length you set, the more VRAM. LoRA 16-bit will use at minimum 4x more VRAM.
* **Continuous fine-tuning is** possible and you can just leave GRPO running in the background.
* In the example notebooks, we use the [**GSM8K dataset**](#gsm8k-reward-functions), the current most popular choice for R1-style training.
* If you’re using a base model, ensure you have a chat template.
* The more you train with GRPO the better. The best part of GRPO is you don't even need that much data. All you need is a great reward function/verifier and the more time spent training, the better your model will get. Expect your reward vs step to increase as time progresses like this:

  <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e44683faa4765a3b803edd4c02c4b468e45cc91d%2Funnamed.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>
* Training loss tracking for GRPO is now built directly into Unsloth, eliminating the need for external tools like wandb etc. It contains full logging details for all reward functions now including the total aggregated reward function itself.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-58d958e1a3bfd968f1b1a4995a28261aa6413337%2FScreenshot%202025-02-20%20at%2004-52-52%20Copy%20of%20Yet%20another%20copy%20of%20Llama3.1_(8B)-GRPO.ipynb%20-%20Colab.png?alt=media" alt=""><figcaption></figcaption></figure>

### RL on unsupported models:

You can also run RL with Unsloth on models that are not supported by vLLM, such as [Qwen3.5](/docs/models/qwen3.5/fine-tune). Simply set `fast_inference=False` when loading the model.

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3.5-4B",
    fast_inference=False,
)
```

## :clipboard:Reward Functions / Verifiers

In Reinforcement Learning a **Reward Function** and a **Verifier** serve distinct roles in evaluating a model’s output. In general, you could interpret them as the same thing however, technically they're not but it does not matter as much as they are usually used in conjunction with each other.

**Verifier**:

* Determines whether the generated response is correct or incorrect.
* It does not assign a numerical score—it simply verifies correctness.
* Example: If a model generates "5" for "2+2", the verifier checks and labels it as "wrong" (since the correct answer is 4).
* Verifiers can also execute code (e.g., in Python) to validate logic, syntax, and correctness without needing manual evaluation.

**Reward Function**:

* Converts verification results (or other criteria) into a numerical score.
* Example: If an answer is wrong, it might assign a penalty (-1, -2, etc.), while a correct answer could get a positive score (+1, +2).
* It can also penalize based on criteria beyond correctness, such as excessive length or poor readability.

**Key Differences**:

* A **Verifier** checks correctness but doesn’t score.
* A **Reward Function** assigns a score but doesn’t necessarily verify correctness itself.
* A Reward Function *can* use a Verifier, but they are technically not the same.

### **Understanding Reward Functions**

GRPO's primary goal is to maximize reward and learn how an answer was derived, rather than simply memorizing and reproducing responses from its training data.

* With every training step, GRPO **adjusts model weights** to maximize the reward. This process fine-tunes the model incrementally.
* **Regular fine-tuning** (without GRPO) only **maximizes next-word prediction probability** but does not optimize for a reward. GRPO **optimizes for a reward function** rather than just predicting the next word.
* You can **reuse data** across multiple epochs.
* **Default reward functions** can be predefined to be used on a wide array of use cases or you can ask ChatGPT/local model to generate them for you.
* There’s no single correct way to design reward functions or verifiers - the possibilities are endless. However, they must be well-designed and meaningful, as poorly crafted rewards can unintentionally degrade model performance.

### :coin:Reward Function Examples

You can refer to the examples below. You can input your generations into an LLM like ChatGPT 4o or Llama 3.1 (8B) and design a reward function and verifier to evaluate it. For example, feed your generations into a LLM of your choice and set a rule: "If the answer sounds too robotic, deduct 3 points." This helps refine outputs based on quality criteria

#### **Example #1: Simple Arithmetic Task**

* **Question:** `"2 + 2"`
* **Answer:** `"4"`
* **Reward Function 1:**
  * If a number is detected → **+1**
  * If no number is detected → **-1**
* **Reward Function 2:**
  * If the number matches the correct answer → **+3**
  * If incorrect → **-3**
* **Total Reward:** *Sum of all reward functions*

#### **Example #2: Email Automation Task**

* **Question:** Inbound email
* **Answer:** Outbound email
* **Reward Functions:**
  * If the answer contains a required keyword → **+1**
  * If the answer exactly matches the ideal response → **+1**
  * If the response is too long → **-1**
  * If the recipient's name is included → **+1**
  * If a signature block (phone, email, address) is present → **+1**

### Unsloth Proximity-Based Reward Function

If you’ve checked out our [**Advanced GRPO Colab Notebook**](#grpo-notebooks), you’ll notice we’ve created a **custom proximity-based reward function** built completely from scratch, which is designed to reward answers that are closer to the correct one. This flexible function can be applied across a wide range of tasks.

* In our examples, we enable reasoning in Qwen3 (Base) and guide it toward specific tasks
* Apply Pre-finetuning strategies to avoid GRPO’s default tendency to just learn formatting
* Boost evaluation accuracy with regex-based matching
* Create custom GRPO templates beyond generic prompts like `think`, e.g., `<start_working_out></end_working_out>`
* Apply proximity-based scoring — models get more reward for closer answers (e.g., predicting 9 instead of 10 is better than 3) while outliers are penalized

#### GSM8K Reward Functions

In our other examples, we use existing GSM8K reward functions by [@willccbb](https://x.com/willccbb) which is popular and shown to be quite effective:

* **correctness\_reward\_func** – Rewards exact label matches.
* **int\_reward\_func** – Encourages integer-only answers.
* **soft\_format\_reward\_func** – Checks structure but allows minor newline mismatches.
* **strict\_format\_reward\_func** – Ensures response structure matches the prompt, including newlines.
* **xmlcount\_reward\_func** – Ensures exactly one of each XML tag in the response.

## :abacus:Using vLLM

You can now use [vLLM](https://github.com/vllm-project/vllm/) directly in your finetuning stack, which allows for much more throughput and allows you to finetune and do inference on the model at the same time! On 1x A100 40GB, expect 4000 tokens / s or so with Unsloth’s dynamic 4bit quant of Llama 3.2 3B Instruct. On a 16GB Tesla T4 (free Colab GPU), you can get 300 tokens / s.\
\
We also magically removed double memory usage when loading vLLM and Unsloth together, allowing for savings of 5GB or so for Llama 3.1 8B and 3GB for Llama 3.2 3B. Unsloth could originally finetune Llama 3.3 70B Instruct in 1x 48GB GPU with Llama 3.3 70B weights taking 40GB of VRAM. If we do not remove double memory usage, then we’ll need >= 80GB of VRAM when loading Unsloth and vLLM together.\
\
But with Unsloth, you can still finetune and get the benefits of fast inference in one package in under 48GB of VRAM! To use fast inference, first install vllm, and instantiate Unsloth with fast\_inference:

```python
# pip install unsloth vllm
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.2-3B-Instruct",
    fast_inference = True,
)
```

## :white\_check\_mark:GRPO Requirement Guidelines

When you’re using Unsloth to do GRPO, we smartly reduce VRAM usage by over 90% when compared to standard implementations with Flash Attention 2 by using multiple tricks! On 20K context lengths for example with 8 generations per prompt, Unsloth uses only **54.3GB of VRAM for Llama 3.1 8B**, whilst standard implementations take **510.8GB (90% less for Unsloth)**.

1. For GRPO's **GPU VRAM requirements for QLoRA 4-bit**, the general rule is the model parameters = the amount of VRAM you will need (you can use less VRAM but this just to be safe). The more context length you set, the more VRAM. LoRA 16-bit will use at minimum 4x more VRAM.
2. Our new memory efficient linear kernels for GRPO slashes memory usage by 8x or more. This shaves 68.5GB of memory, whilst being actually faster through the help of torch.compile!
3. We leverage our smart [Unsloth gradient checkpointing](https://unsloth.ai/blog/long-context) algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves 52GB of memory.
4. Unsloth also uses the same GPU / CUDA memory space as the underlying inference engine (vLLM), unlike implementations in other packages. This shaves 16GB of memory.

| Metrics                                        | Unsloth            | Standard + FA2 |
| ---------------------------------------------- | ------------------ | -------------- |
| Training Memory Cost (GB)                      | 42GB               | 414GB          |
| GRPO Memory Cost (GB)                          | 9.8GB              | 78.3GB         |
| Inference Cost (GB)                            | 0GB                | 16GB           |
| Inference KV Cache for 20K context length (GB) | 2.5GB              | 2.5GB          |
| Total Memory Usage                             | 54.33GB (90% less) | 510.8GB        |

In typical standard GRPO implementations, you need to create 2 logits of size (8. 20K) to calculate the GRPO loss. This takes 2 \* 2 bytes \* 8 (num generations) \* 20K (context length) \* 128256 (vocabulary size) = 78.3GB in VRAM.

Unsloth shaves 8x memory usage for long context GRPO, so we need only an extra 9.8GB in extra VRAM for 20K context lengths!

We also need to from the KV Cache in 16bit. Llama 3.1 8B has 32 layers, and both K and V are 1024 in size. So memory usage for 20K context length = 2 \* 2 bytes \* 32 layers \* 20K context length \* 1024 = 2.5GB per batch. We would set the batch size for vLLM to 8, but we shall leave it at 1 for our calculations to save VRAM. Otherwise you will need 20GB for the KV cache.

## 🎥 Unsloth RL 3 hour Workshop Video

{% embed url="<https://www.youtube.com/watch?v=OkEGJ5G3foU>" %}

## :mortar\_board:Further Reading

1. Nathan Lambert's RLHF Book is a must! <https://rlhfbook.com/c/11-policy-gradients.html>
2. Yannic Kilcher's GRPO Youtube video is also a must! <https://www.youtube.com/watch?v=bAWV_yrqx4w>
3. We did a 3 hour workshop at AI Engineer World's Fair 2025. Slides are other material are at <https://docs.unsloth.ai/ai-engineers-2025>
4. Advanced GRPO notebook via Unsloth. <https://docs.unsloth.ai/basics/reinforcement-learning-guide/tutorial-train-your-own-reasoning-model-with-grpo>
5. GRPO from a base model notebook: <https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb>

## Video Tutorials

Here are some video tutorials created by amazing YouTubers who we think are fantastic!

{% embed url="<https://www.youtube.com/watch?v=9t-BAjzBWj8>" %}

{% columns %}
{% column width="50%" %}
{% embed url="<https://www.youtube.com/watch?t=3289s&v=bbFEYPx9Hpo>" %}
Great to learn about how to prep your dataset and explanations behind Reinforcement Learning + GRPO basics
{% endembed %}

{% embed url="<https://www.youtube.com/watch?v=oF0_eMhzRaQ>" %}
{% endcolumn %}

{% column width="50%" %}
{% embed url="<https://www.youtube.com/watch?v=juOh1afy-IE>" %}

{% embed url="<https://www.youtube.com/watch?v=SoPE1cUz3Hs>" %}
Local GRPO on your own device
{% endembed %}
{% endcolumn %}
{% endcolumns %}


# Reinforcement Learning GRPO with 7x Longer Context

Learn how Unsloth enables ultra long context RL fine-tuning.

Reinforcement learning's (RL) biggest challenge is supporting long reasoning traces. We're introducing new batching algorithms to enable \~**7x longer context** (can be more than 12x) RL training with no accuracy or speed degradation vs. other optimized setups that use FA3, kernels & chunked losses.

* Unsloth now trains gpt-oss QLoRA with **380K context** on a single 192GB NVIDIA B200 GPU
* [Qwen3](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth)-8B GRPO reaches **110K context** on an 80GB VRAM H100 via [vLLM](#vllm-for-rl) and QLoRA, and **65K** for [gpt-oss](/docs/models/gpt-oss-how-to-run-and-fine-tune/gpt-oss-reinforcement-learning) with BF16 LoRA.
* On 24GB VRAM, gpt-oss reaches 20K context and 32K for [Qwen3-VL](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune)-8B QLoRA
* Unsloth GRPO RL runs with Llama, Gemma & all models auto support longer contexts

Our new data-movement and batching kernels and algorithms unlocks more context by:

* Dynamic [flattened sequence chunking](#flattened-sequence-length-chunking) to avoid materializing massive logit tensors and
* [Offloading log softmax](#offloading-activations-for-log-softmax) activations which prevents silent memory growth over time.

{% hint style="info" %}
**You can combine all features in Unsloth together:**

1. Unsloth's [weight-sharing](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl) feature with [vLLM](https://github.com/vllm-project/vllm) and our Standby Feature in [Memory Efficient RL](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl)
2. Unsloth's [Flex Attention](/docs/models/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) for long context gpt-oss and our [500K Context Training](/docs/blog/500k-context-length-fine-tuning)
3. Float8 training in [FP8 RL](/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning) and Unsloth's [async gradient checkpointing](https://unsloth.ai/blog/long-context) and much more
   {% endhint %}

### :tada:Getting started

To get started, you can use any existing [GRPO notebooks](/docs/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks) (or update Unsloth if local):

{% columns %}
{% column width="33.33333333333333%" %}
[**gpt-oss-20b**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) GSPO

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb>" %}
{% endcolumn %}

{% column width="33.33333333333333%" %}
[**Qwen3-VL-8B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb) Vision RL

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb>" %}
{% endcolumn %}

{% column width="33.33333333333333%" %}
[Qwen3-8B - **FP8**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) L4 GPU

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

Adopting Unsloth for your RL tasks provides a robust framework for managing large-scale models efficiently. To effectively utilize Unsloth's enhancements:

* **Hardware Recommendations**: Use of NVIDIA H100 or equivalent for optimal VRAM utilization.
* **Configuration Tips**: Ensure `batch_size` and `gradient_accumulation_steps` settings align with your computational resources for best performance.

{% hint style="success" %}
Update Unsloth to the latest Pypi release to get the latest updates:

```
pip install --upgrade --no-cache-dir unsloth unsloth_zoo
```

{% endhint %}

Our benchmarks highlight the memory savings achieved in comparison to earlier versions for GPT OSS and Qwen3-8B. Both plots below (without [standby](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl)) were run with `batch_size = 4` and `gradient_accumulation_steps=2` , since standby by design uses all VRAM.

For our benchmarks, we compare BF16 GRPO to Hugging Face with all optimizations enabled (all kernels in kernels library, Flash Attention 3, chunked loss kernels, etc):

### :1234:Flattened sequence length chunking

Previously, Unsloth reduced memory usage of RL by avoiding the full materialization of the logits tensor through chunking over the batch dimension. A rough estimate of the VRAM required to materialize logits during the forward pass is shown in Equation (1).

$$
\text{Equation 1: } \text{Logit Memory (GB)} = \frac{\text{batch size} \times\text{context length} \times \text{vocab dim}}{1024^3}
$$

Using this formulation, a configuration with `batch_size = 4`, `context_length = 8192`, and `vocab_dim = 128,000` would require approximately **3.3 GB of VRAM** to store the logits tensor.

Via [Long Context gpt-oss](/docs/models/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) last year, we then introduced a fused loss approach for GRPO. This approach ensures that only a single batch sample is processed at a time, significantly reducing peak memory usage. Under the same configuration, VRAM usage drops to approximately **0.83 GB**, as reflected in Equation (2).

$$
\text{Equation 2: }\text{Logit Memory (GB)} = \frac{\text{context length} \times \text{vocab dim}}{1024^3}
$$

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fy1TkzxySrNAeeSWJSVLU%2Funsloth_vs_trl_gpt_oss.png?alt=media&amp;token=0303423d-1454-4410-8be8-7d6110ac1df0" alt="" width="375"><figcaption><p>Figure 1: gpt-oss BF16 GRPO LoRA (Unsloth vs. HF with all optimizations on)</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmfKhenN0TGRDlMcuxob6%2Fqwen38b%20long%20context%20grpo.png?alt=media&amp;token=22883f90-5bf0-4478-91a9-6a191c920f12" alt="" width="375"><figcaption><p>Figure 2: Qwen3-8B QLoRA GRPO LoRA (Unsloth vs. HF with all optimizations on)</p></figcaption></figure></div>

In this update, we extend the same idea further by introducing chunking across the **sequence dimension** as well. Instead of materializing logits for the entire `(batch_size × context_length)` space at once, we flatten these dimensions and process them in smaller chunks using a configurable multiplier. This allows Unsloth to support substantially longer contexts without increasing peak memory usage.

In Figure 5 below, we use a multiplier of `max(4, context_length // 4096)`, though any multiplier can be specified depending on the desired memory–performance tradeoff. With this setting, the same example configuration (`batch_size = 4`, `context_length = 8192`, `vocab_dim = 128,000`) now requires only **0.207 GB of VRAM** for logits materialization.

$$
\text{Equation 3: }\text{Logit Memory (GB)} = \frac{\frac{\text{context length}}{\text{multiplier}} \times \text{vocab dim}}{1024^3}
$$

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FWFTbejdIn3T6E6yHgF1Z%2FCode_Generated_Image%20(2).png?alt=media&amp;token=790a1ee4-2814-4b29-afcb-bb9ffd1eb729" alt="" width="375"><figcaption><p>Figure 3: gpt-oss-20b (H100) Unsloth new vs. old</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fi4QipufoavtPKyeRU0Vv%2FCode_Generated_Image%20(3).png?alt=media&amp;token=226c5a3c-a0a4-458d-a0df-8c84523b04b5" alt="" width="375"><figcaption><p>Figure 4: Qwen3-8B (H100) Unsloth new vs. old</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkSIh5DIWvKGemnNHowPs%2FCode_Generated_Image_4.png?alt=media&amp;token=0a3dfe85-ae8c-4280-bc0a-6c1f1523c90e" alt="" width="375"><figcaption><p>Figure 5: gpt-oss-20b (H100)</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRP1RPiOeIYOt82L1Ifkc%2FCode_Generated_Image_5.png?alt=media&amp;token=4ce06b0f-2464-41fd-8795-e5bf0dbf4327" alt="" width="375"><figcaption><p>Figure 6: Qwen3-8B (B200)</p></figcaption></figure></div>

This update is reflected in the compiled `chunked_hidden_states_selective_log_softmax` below, which now supports chunking across both the batch and sequence dimensions. To preserve the logits tensor (`[batch_size, context_length, vocab_dim]`), it is always chunked across the batch dimension. Additional sequence chunking is controlled via `unsloth_logit_chunk_multiplier` in the GRPO configuration; if unset, it defaults to `max(4, context_length // 4096)`. In the example below, `input_ids_chunk[0]` corresponds to the size of the hidden states mini batches in optimization 2.

```python
logprobs_chunk = chunked_hidden_states_selective_log_softmax(
    new_hidden_states_chunk, 
    lm_head, 
    completion_ids, 
    chunks=input_ids_chunk.shape[0]*multiplier, 
    logit_scale_multiply=logit_scale_multiply,
    logit_scale_divide=logit_scale_divide,
    logit_softcapping=logit_softcapping,
    temperature=temperature,                
)
```

1. We utilize torch.compile with custom compile options to reduce VRAM and increase speed.
2. All chunked logits are upcasted in float32 to preserve accuracy.
3. We support logit softcapping, temperature scaling and all other features.

### :ghost:Hidden States Chunking

We also observed that at longer context lengths, hidden states can become a significant contributor to memory usage. For demonstration, we will assume `hidden_states_dim=4096`. The corresponding memory usage follows a similar formulation to the logits case, shown below.&#x20;

$$
\text{Hidden States Memory (GB)} = \frac{\text{batch size} \times\text{context length} \times \text{hidden states dim}}{1024^3}
$$

With a `batch_size = 8` and `context_length = 64000`, this would result in a VRAM usage of approximately **2 GB**. In this release, we introduce optional chunking over the batch dimension for the hidden states tensor during log-probability computation. This would cause the VRAM usage to be divided by the batch size or in this case be **0.244 GB**.This reduces the peak VRAM required to materialize hidden states, as reflected in the updated equation below:

$$
\text{Hidden States Memory (GB)} = \frac{\text{context length} \times \text{hidden states dim}}{1024^3}
$$

Similar to our cross entropy loss in our [500K Context Training](/docs/blog/500k-context-length-fine-tuning) release, the new implementation **automatically tunes hidden state batching**. Users can also control this behavior via `unsloth_grpo_mini_batch`. However, increasing `unsloth_grpo_mini_batch` beyond the optimal value can introduce slight performance increase or slowdown (usually faster) compared to the previous loss function.

However, during a GPT-OSS run (`context_length = 8192, batch_size = 4, gradient_accumulation_steps = 2`), setting `unsloth_grpo_mini_batch = 1` and `unsloth_logit_chunk_multiplier = 4` results in **little to no speed degradation while reducing VRAM usage by approximately 5 GB** compared to older versions of Unsloth.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYZsgoKpZKyJKrNmbvehR%2FCode_Generated_Image%20(4).png?alt=media&amp;token=5d3c0605-9ed8-4d4a-a722-a85132510222" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="success" %}
**Note:** In Figures 3 and 4, we use the maximum effective batch size, which is 8 in this setup. The effective batch size is computed as `batch_size × gradient_accumulation_steps`, giving `4 × 2 = 8`. For a deeper explanation of how effective batch sizes work in RL, see our [advanced RL documentation](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation).&#x20;
{% endhint %}

### :cactus:Offloading activations for log softmax

During the development of this release, we discovered that when tiling across the batch dimension for hidden states, the activations were not being offloaded after the fused logits and logprobs computation. Because logits are computed one batch at a time using `hidden_states[i] @ lm_head`, the existing activation offloading and gradient checkpointing logic, designed to operate within the model’s forward pass did not apply in this case.

To address this, we added explicit logic to offload these activations outside the model’s forward pass, as shown in the Python pseudocode below:

```python
class Unsloth_Offloaded_Log_Softmax(torch.autograd.Function):
    def forward(...):
        with torch.no_grad():
            output = chunked_hidden_states_selective_log_softmax(hidden_states, lm_head, ...)
        return output
    def backward(ctx, grad_output):
        hidden_states = ctx.saved_hidden_states
        hidden_states.requires_grad_(True)
        with torch.enable_grad():
            output = chunked_hidden_states_selective_log_softmax(hidden_states, lm_head, ...)
        torch.autograd.backward(output, grad_output)
        return ...
```

{% hint style="success" %}
**Note:** This feature is only effective when chunking across the batch dimension or when `unsloth_grpo_mini_batch > 1`. If all hidden states are materialized at once during the forward pass (i.e., `unsloth_grpo_mini_batch = 1`), the backward pass requires the same amount of memory in the GPU regardless of whether activations are offloaded. Since activation offloading introduces a slight performance slowdown without reducing memory usage in this case, it provides no benefit.
{% endhint %}

### :sparkles:Configuring parameters:

If you do not configure `unsloth_grpo_mini_batch` and `unsloth_logit_chunk_multiplier`, we will **automatically tune these two parameters** for you based on your available VRAM and depending on the size of your context length. Below however is how you can change these variables in your GRPO run:

```python
training_args = GRPOConfig(
    ...
    unsloth_grpo_mini_batch = 3
    unsloth_logit_chunk_multiplier = 2
    ...
)
```

A visualization of the optimizations and `unsloth_grpo_mini_batch` and `unsloth_logit_chunk_multiplier` can be seen in the diagram below.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F2OmZA297HzG3CdRzi3X5%2FLogit%20Chunking%20(1).png?alt=media&amp;token=b953a62b-fefa-43f2-a9ce-108675b8735f" alt="" width="375"><figcaption></figcaption></figure>

The 3 matrices represent the overall larger batch or `unsloth_grpo_mini_batch` (represented by the number of black brackets) and the rows of each of the matrices represents the context length that the `unsloth_logit_chunk_multiplier`  chunks the sequence length by (represented by the number of red brackets).&#x20;

### :vhs:vLLM for RL

**For RL workflows, the inference/generation phase is the main bottleneck**. To address this, we utilize [vLLM](https://github.com/vllm-project/vllm), which has accelerated generation by up to 11x compared to normal generation. Since GRPO was popularized last year, vLLM has been a core component of most RL frameworks including Unsloth. We want to extend our gratitude to the vLLM team and all its contributors for their work as they play a pivotal role in making Unsloth’s RL better!

To try longer context RL, you can use any existing [GRPO notebooks](/docs/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks) (or update Unsloth if local):

{% columns %}
{% column width="33.33333333333333%" %}
[**gpt-oss-20b**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) - GSPO

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb>" %}
{% endcolumn %}

{% column width="33.33333333333333%" %}
[**Qwen3-VL-8B**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb) Vision RL

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb>" %}
{% endcolumn %}

{% column width="33.33333333333333%" %}
[Qwen3-8B - **FP8**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) L4 GPU

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

Acknowledgements: A huge thank you to the Hugging Face team and libraries for powering Unsloth and making this possible.


# Vision Reinforcement Learning (VLM RL)

Train Vision/multimodal models via GRPO and RL with Unsloth!

Unsloth now supports vision/multimodal RL with [Qwen3-VL](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune), [Gemma 3](/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune) and more. Due to Unsloth's unique [weight sharing](/docs/get-started/reinforcement-learning-rl-guide#what-unsloth-offers-for-rl) and custom kernels, Unsloth makes VLM RL **1.5–2× faster,** uses **90% less VRAM**, and enables **15× longer context** lengths than FA2 setups, with no accuracy loss. This update also introduces Qwen's [GSPO](#gspo-rl) algorithm.

Unsloth can train Qwen3-VL-8B with GSPO/GRPO on a free Colab T4 GPU. Other VLMs work too, but may need larger GPUs. Gemma requires newer GPUs than T4 because vLLM [restricts to Bfloat16](/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune#unsloth-fine-tuning-fixes), thus we recommend NVIDIA L4 on Colab. Our notebooks solve numerical math problems involving images and diagrams:

* **Qwen-3 VL-8B** (vLLM inference)**:** [Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb)
* **Qwen-2.5 VL-7B** (vLLM inference)**:** [Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb) •[ Kaggle](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen2_5_7B_VL_GRPO.ipynb\&accelerator=nvidiaTeslaT4)
* **Gemma-3-4B** (Unsloth inference): [Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision-GRPO.ipynb)

We have also added vLLM VLM integration into Unsloth natively, so all you have to do to use vLLM inference is enable the `fast_inference=True` flag when initializing the model. Special thanks to [Sinoué GAD](https://github.com/unslothai/unsloth/pull/2752) for providing the [first notebook](https://github.com/GAD-cell/vlm-grpo/blob/main/examples/VLM_GRPO_basic_example.ipynb) that made integrating VLM RL easier!

This VLM support also integrates our latest update for even more memory efficient + faster RL including our [Standby feature](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl#unsloth-standby), which uniquely limits speed degradation compared to other implementations.

{% hint style="info" %}
You can only use `fast_inference` for VLMs supported by vLLM. Some models, like Llama 3.2 Vision thus only can run without vLLM, but they still work in Unsloth.
{% endhint %}

```python
os.environ['UNSLOTH_VLLM_STANDBY'] = '1' # To enable memory efficient GRPO with vLLM
model, tokenizer = FastVisionModel.from_pretrained(
    model_name = "Qwen/Qwen2.5-VL-7B-Instruct",
    max_seq_length = 16384, #Must be this large to fit image in context
    load_in_4bit = True, # False for LoRA 16bit
    fast_inference = True, # Enable vLLM fast inference
    gpu_memory_utilization = 0.8, # Reduce if out of memory
)
```

It is also important to note, that vLLM does not support LoRA for vision/encoder layers, thus set `finetune_vision_layers = False` when loading a LoRA adapter.\
However you CAN train the vision layers as well if you use inference via transformers/Unsloth.

```python
# Add LoRA adapter to the model for parameter efficient fine tuning
model = FastVisionModel.get_peft_model(
    model,

    finetune_vision_layers     = False,# fast_inference doesn't support finetune_vision_layers yet :(
    finetune_language_layers   = True, # False if not finetuning language layers
    finetune_attention_modules = True, # False if not finetuning attention layers
    finetune_mlp_modules       = True, # False if not finetuning MLP layers

    r = lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    lora_alpha = lora_rank*2, # *2 speeds up training
    use_gradient_checkpointing = "unsloth", # Reduces memory usage
    random_state = 3407,
)
```

## :butterfly:Qwen 2.5 VL Vision RL Issues and Quirks

During RL for Qwen 2.5 VL, you might see the following inference output:

{% code overflow="wrap" %}

```
 addCriterion
 <tool_call>\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n\n addCriterion\n\n 自动生成\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n addCriterion\n\n\n addCriterion\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
```

{% endcode %}

This was [reported](https://github.com/QwenLM/Qwen2.5-VL/issues/759) as well in Qwen2.5-VL-7B-Instruct output unexpected results "addCriterion". In fact we see this as well! We tried both non Unsloth, bfloat16 and float16 machines and other things, but it appears still. For example item 165 ie `train_dataset[165]` from the [AI4Math/MathVista](https://huggingface.co/datasets/AI4Math/MathVista) dataset is below:

{% code overflow="wrap" %}

```
Figure is an overhead view of the path taken by a race car driver as his car collides with the racetrack wall. Just before the collision, he is traveling at speed $v_i=70 \mathrm{~m} / \mathrm{s}$ along a straight line at $30^{\circ}$ from the wall. Just after the collision, he is traveling at speed $v_f=50 \mathrm{~m} / \mathrm{s}$ along a straight line at $10^{\circ}$ from the wall. His mass $m$ is $80 \mathrm{~kg}$. The collision lasts for $14 \mathrm{~ms}$. What is the magnitude of the average force on the driver during the collision?
```

{% endcode %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-61a659529171fcc10ed6398a15912b21d6b1a076%2FUntitled.png?alt=media" alt="" width="128"><figcaption></figcaption></figure>

And then we get the above gibberish output. One could add a reward function to penalize the addition of addCriterion, or penalize gibberish outputs. However, the other approach is to train it for longer. For example only after 60 steps ish do we see the model actually learning via RL:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-5f34f66f0ac6508fd28343b16592c59b889ec5ca%2Fimage.webp?alt=media" alt=""><figcaption></figcaption></figure>

{% hint style="success" %}
Forcing `<|assistant|>` during generation will reduce the occurrences of these gibberish results as expected since this is an Instruct model, however it's still best to add a reward function to penalize bad generations, as described in the next section.
{% endhint %}

## :medal:Reward Functions to reduce gibberish

To penalize `addCriterion` and gibberish outputs, we edited the reward function to penalize too much of `addCriterion` and newlines.

```python
def formatting_reward_func(completions,**kwargs):
    import re
    thinking_pattern = f'{REASONING_START}(.*?){REASONING_END}'
    answer_pattern = f'{SOLUTION_START}(.*?){SOLUTION_END}'

    scores = []
    for completion in completions:
        score = 0
        thinking_matches = re.findall(thinking_pattern, completion, re.DOTALL)
        answer_matches = re.findall(answer_pattern, completion, re.DOTALL)
        if len(thinking_matches) == 1:
            score += 1.0
        if len(answer_matches) == 1:
            score += 1.0

        # Fix up addCriterion issues
        # See https://docs.unsloth.ai/new/vision-reinforcement-learning-vlm-rl#qwen-2.5-vl-vision-rl-issues-and-quirks
        # Penalize on excessive addCriterion and newlines
        if len(completion) != 0:
            removal = completion.replace("addCriterion", "").replace("\n", "")
            if (len(completion)-len(removal))/len(completion) >= 0.5:
                score -= 2.0

        scores.append(score)
    return scores
```

## :checkered\_flag:GSPO Reinforcement Learning

This update in addition adds GSPO ([Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071)) which is a variant of GRPO made by the Qwen team at Alibaba. They noticed that GRPO implicitly results in importance weights for each token, even though explicitly advantages do not scale or change with each token.

This lead to the creation of GSPO, which now assigns the importance on the sequence likelihood rather than the individual token likelihoods of the tokens. The difference between these two algorithms can be seen below, both from the GSPO paper from Qwen and Alibaba:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-45d743dd5dcd590626777ce09cfab61808aa8c24%2Fimage.png?alt=media" alt="" width="563"><figcaption><p>GRPO Algorithm, Source: <a href="https://arxiv.org/abs/2507.18071">Qwen</a></p></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-ee755850cbe17482ce240dde227d55c62e9a3e64%2Fimage.png?alt=media" alt="" width="563"><figcaption><p>GSPO algorithm, Source: <a href="https://arxiv.org/abs/2507.18071">Qwen</a></p></figcaption></figure>

In Equation 1, it can be seen that the advantages scale each of the rows into the token logprobs before that tensor is sumed. Essentially, each token is given the same scaling even though that scaling was given to the entire sequence rather than each individual token. A simple diagram of this can be seen below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-b3c944808a15dde0a7ff45782f9f074993304bf1%2FCopy%20of%20GSPO%20diagram%20(1).jpg?alt=media" alt="" width="286"><figcaption><p>GRPO Logprob Ratio row wise scaled with advantages</p></figcaption></figure>

Equation 2 shows that the logprob ratios for each sequence is summed and exponentiated after the Logprob ratios are computed, and only the resulting now sequence ratios get row wise multiplied by the advantages.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-62fc5b50921e79cce155d2794201c9b96faf941e%2FGSPO%20diagram%20(1).jpg?alt=media" alt="" width="313"><figcaption><p>GSPO Sequence Ratio row wise scaled with advantages</p></figcaption></figure>

Enabling GSPO is simple, all you need to do is set the `importance_sampling_level = "sequence"` flag in the GRPO config.

```python
training_args = GRPOConfig(
    output_dir = "vlm-grpo-unsloth",
    per_device_train_batch_size = 8,
    gradient_accumulation_steps = 4,
    learning_rate = 5e-6,
    adam_beta1 = 0.9,
    adam_beta2 = 0.99,
    weight_decay = 0.1,
    warmup_ratio = 0.1,
    lr_scheduler_type = "cosine",
    optim = "adamw_8bit",
    # beta = 0.00,
    epsilon = 3e-4,
    epsilon_high = 4e-4,
    num_generations = 8,    
    max_prompt_length = 1024,
    max_completion_length = 1024,
    log_completions = False,
    max_grad_norm = 0.1,
    temperature = 0.9,
    # report_to = "none", # Set to "wandb" if you want to log to Weights & Biases
    num_train_epochs = 2, # For a quick test run, increase for full training
    report_to = "none"
    
    # GSPO is below:
    importance_sampling_level = "sequence",
    
    # Dr GRPO / GAPO etc
    loss_type = "dr_grpo",
)
```

Overall, Unsloth now with VLM vLLM fast inference enables for both 90% reduced memory usage but also 1.5-2x faster speed with GRPO and GSPO!

If you'd like to read more about reinforcement learning, check out out RL guide:

[Reinforcement Learning](/docs/get-started/reinforcement-learning-rl-guide)

***Authors:** A huge thank you to* [*Keith*](https://www.linkedin.com/in/keith-truongcao-7bb84a23b/) *and* [*Datta*](https://www.linkedin.com/in/datta0/) *for contributing to this article!*


# FP8 Reinforcement Learning

Train reinforcement learning (RL) and GRPO in FP8 precision with Unsloth.

We're introducing FP8-precision training for RL, making FP8 GRPO now possible on **consumer GPUs** (RTX 40, 50 etc). DeepSeek-R1 demonstrated how powerful FP8 can be and with Unsloth, Qwen3-1.7B FP8 GRPO now works on just **5GB of VRAM**.

Faster RL inference is critical as it's the most compute-intensive workload in RL. We collabed with [TorchAO](https://github.com/pytorch/ao) from PyTorch to enable performance gains with no loss in accuracy.

* **\~1.4× faster** RL inference via [vLLM](https://github.com/vllm-project/vllm) • 2x longer context vs. BF16 and FP16
* **60% less VRAM** and **10× longer** context than other FP8 RL implementations
* Unsloth is the **only framework** to make FP8 RL LoRA work on consumer GPUs (e.g. NVIDIA GeForce RTX 40 and 50 Series). Also works on H100, H200, B200 etc.
* Use `load_in_fp8 = True` within `FastLanguageModel` to enable FP8 RL.
* Though Qwen3-8B fits in 16GB VRAM, free Colab NVIDIA Tesla T4 GPUs **don’t support FP8**. So our notebooks use **24GB L4 GPUs which fits Qwen3-14B**.

**Notebooks:** [Qwen3-8B FP8 GRPO](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb) and [Llama-3.2-1B FP8 GRPO](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama_FP8_GRPO.ipynb)

{% hint style="success" %}
Bonus: You’ll notice Unsloth now uses much less VRAM. We’ll share details in a new blog soon.
{% endhint %}

Our FP8 support uses Unsloth’s [weight-sharing feature](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl), reducing VRAM use by another **50%**, enabling **10× more** context with no accuracy loss. We use [vLLM](https://github.com/vllm-project/vllm) for fast inference and, our techniques like Unsloth [Standby](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl) and [Flex Attention](/docs/models/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) to further reduce VRAM use. TorchAO enables universal on the fly FP8, so Llama, Gemma, Mistral & more work. We’ve also [uploaded](#unsloth-fp8-uploads) most FP8 models (including Qwen3).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNhbi7jRc6zwCAeuddBBk%2Foutput(14).png?alt=media&amp;token=80ad0712-4626-4536-aa57-29bc53b40540" alt="" width="375"><figcaption><p>Reward plot shows FP8 following the same trend as BF16</p></figcaption></figure>

### :sunflower:FP8 vs BF16 Training

Research shows that FP8 training can largely match BF16 accuracy and if you serve models in FP8, **training and serving in the same precision** helps preserve accuracy. Also FP8 vs BF16 yields 1.6x higher throughput on H100s and has 2x lower memory usage.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FApLfXUBVZSbjpPhRJG6z%2Ffp8%20f16%20quant.png?alt=media&amp;token=77a2917a-f191-44a7-8597-6796fcf24ed7" alt="" width="375"><figcaption></figcaption></figure>

#### Weight scales & FP8 types

Quantized training stores a low-precision weight (e.g., FP8) plus a higher-precision scale (FP16/BF16/FP32). You approximately recover the original weight via: `original_weight ≈ quantized_weight * weight_scale`

The scale maps the weight’s range into FP8’s representable range. More scales usually improve accuracy, but scales cost extra high-precision memory, so it’s a tradeoff. [DeepSeek R1](https://arxiv.org/abs/2501.12948), for instance, mostly favors block quantization.

There are 3 common FP8 types as defined by vLLM's [llm-compressor](https://github.com/vllm-project/llm-compressor). We benchmarked Qwen3-8B on all 3 types, and also checked throughput, MMLU Pro and GQPA Diamond. We find **FP8 Block-Wise or Per-Channel (-FP8-Dynamic) is the best** in terms of accuracy and throughput.

<table><thead><tr><th width="121">Type</th><th width="225.20001220703125"></th><th width="126.4000244140625">Throughput</th><th width="121.60003662109375">MMLU Pro</th><th>GQPA Diamond</th></tr></thead><tbody><tr><td></td><td>Bfloat16 Baseline</td><td>11,367</td><td><strong>62.04%</strong></td><td>28.79%</td></tr><tr><td>Block-wise</td><td>Scales per block (128X128)</td><td>12,041</td><td><strong>62.37%</strong></td><td><strong>29.29%</strong></td></tr><tr><td>Per-Channel</td><td>1 scale per row or column</td><td>12,963</td><td>61.89%</td><td><strong>31.82%</strong></td></tr><tr><td>Per-Tensor</td><td>1 scale for the whole tensor</td><td><strong>13,681</strong></td><td>61.83%</td><td>27.78%</td></tr></tbody></table>

### :zap:FP8 Performance Benchmarks

Unsloth FP8 RL inference via vLLM is generally 1.4x faster than BF16. You may see even more speed improvements if the model is larger!

#### Accuracy Training loss Benchmarks

We tested multiple models including Qwen3-4B, 8B, 14B, Llama 3.2 1B, 3B, Qwen3-VL-2B, Qwen3-VL 4B and many more. All were trained both in BF16 and FP8. As seen in the plots, the **loss curves during SFT for BF16 and FP8 closely track each other**. There isn’t much to choose between the two data types in terms of training loss:

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FR6Hx9RtgqPXnYxvx5BbR%2FW%26B%20Chart%2025_11_2025%2C%208_54_56%20am.png?alt=media&amp;token=d1d70d59-df00-45bb-8352-e833f9b5f3cd" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlUzs2uNkCyF1ulNdrVRc%2FW%26B%20Chart%2025_11_2025%2C%208_56_50%20am.png?alt=media&amp;token=09545235-c9fa-4b76-a834-ffe0ceb8f639" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

For GRPO specifically, due to generation differences, the goal is to see if the reward plots at least match up and not diverge (sometimes for eg Qwen3-14B runs might not be exactly similar)

{% columns %}
{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FeLBs5GrQb988GcrYVzpF%2FW%26B%20Chart%2025_11_2025%2C%209_00_50%20am.png?alt=media&amp;token=59220833-33c6-4c28-abe7-b5d0d93a0a17" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FPqXVeofauAIr5Qngm9d2%2FW%26B%20Chart%2025_11_2025%2C%209_08_06%20am.png?alt=media&amp;token=16498cf1-17e1-4984-b933-fe3633e19a6b" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FC76ql9G59SB0v3nG3pbL%2FW%26B%20Chart%2025_11_2025%2C%209_05_32%20am.png?alt=media&amp;token=554b6fe8-c121-48a4-8b33-41f28fc38ebb" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FqM5NKHjOxqJv0hrzmr2B%2FW%26B%20Chart%2025_11_2025%2C%209_07_12%20am.png?alt=media&amp;token=a7ad9eb0-0ea2-4364-982a-0875ec63459f" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### :shinto\_shrine:Inference = 96% of RL training

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FTvC7GqMM5XAfV8Zv2tpf%2Fimage.avif?alt=media&amp;token=62b40c34-3111-40a9-b02a-4bfa8826402d" alt=""><figcaption></figcaption></figure>

In RL, we have to call the LLM / VLM to generate some possible candidate solutions to some run, then we score each possible solution and **reward good solutions, and penalize bad answers**. To achieve maximum efficiency, we must make inference nearly 100% of the training run. In Unsloth, we **managed to make training take only <4% of the entire RL run, with 96% being purely vLLM inference.**

For example for Qwen-3-8B, which is 1.15x faster on shorter sequence lengths, vLLM FP8 itself for inference (without training) throughput is also 1.15x faster. We see our RL run in Unsloth attains also 1.15x faster on tokens processed, showing how **training overhead is negligible in Unsloth.**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F105iKEPXAor00mdUPTfo%2FTokens%20Processed%20during%20RL.svg?alt=media&amp;token=ca1c1d76-64b2-4019-91ac-7043f0ab79fd" alt=""><figcaption></figcaption></figure>

### :1234:60% less memory usage

In theory, you’d expect memory savings to roughly **equal to the model’s weight memory**, because: optimizer states are still stored in high precision and activations are also stored in high precision (for now). Our findings match the theory. For LoRA fine-tuning, we observed: **\~30 GB saved** for **Qwen3-32B, \~14 GB saved** for **Qwen2.5-14B** and **\~8 GB saved** for **Qwen3-8B**

For **BF16 LoRA fine-tuning on** Qwen3-32B, we were ooming at higher batch sizes and had to shrink the batch. The **FP8 variant had no such issues**, and we could use **larger batch sizes** without OOMing.

Also reminder in Unsloth we share vLLM's memory space for the weights as introduced in [Memory Efficient RL](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl) - we have bought this trick over to the FP8 domain!

| 80GB GPU                                                                                                                                                            | Inference Engine   | Training Engine                          |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ---------------------------------------- |
| Model Weights                                                                                                                                                       | **8GB SHARED FP8** | **<<< SHARED**                           |
| <p><mark style="background-color:purple;"><strong>Multi-purpose</strong></mark></p><p><mark style="background-color:purple;"><strong>72GB space</strong></mark></p> | KV Cache           | Activations, Gradients, Optimizer States |

To enable [Unsloth Standby](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl) for FP8 (or BF16) RL, simply add the below to all RL / GRPO training runs before any Unsloth import:

```python
import os
os.environ["UNSLOTH_VLLM_STANDBY"] = "1"
```

### :question:How to use FP8 RL / installation

Simply update Unsloth or install Unsloth in a new virtual environment for H100, L4, RTX 50x, RTX 40x, H200s, B200s, and any NVIDIA GPU (consumer or data center grade) released after the RTX 4090.

To update Unsloth: `pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth unsloth_zoo`Or make a new environment:

{% code overflow="wrap" %}

```bash
python -m venv unsloth_env
source unsloth_env/bin/activate

pip install unsloth vllm
pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu128 --force-reinstall
pip install --pre fbgemm-gpu fbgemm-gpu-genai --index-url https://download.pytorch.org/whl/cu128 --force-reinstall
pip install --upgrade numba numpy
```

{% endcode %}

Then use `load_in_fp8 = True` and you're good to go! We'll auto map the model name to the Float8 variant, or we'll on the fly convert the model to Float8!

<pre class="language-python" data-overflow="wrap"><code class="lang-python">import os
os.environ['UNSLOTH_VLLM_STANDBY'] = "1" # Unsloth standby saves 30%+ memory for RL
from unsloth import FastLanguageModel
import torch
max_seq_length = 2048 # Can increase for longer reasoning traces
lora_rank = 32 # Larger rank = smarter, but slower
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-8B",
    max_seq_length = max_seq_length,
    load_in_4bit = False, # False for LoRA 16bit
    fast_inference = True, # Enable vLLM fast inference
    max_lora_rank = lora_rank,
<strong>    load_in_fp8 = True, # Float8 RL / GRPO!
</strong>)
</code></pre>

For example on a RTX 5090 (reminder to set `os.environ["UNSLOTH_VLLM_STANDBY"] = "1"` )

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlVA3v7E5J8pHb1QKLi2V%2Fimage.png?alt=media&amp;token=20b5329c-6ac2-479a-a4cc-2a0d74486696" alt="" width="375"><figcaption></figcaption></figure>

Then use our 2 FP8 notebooks for RL:

{% columns %}
{% column %}
**Qwen3-8B FP8 RL Colab**

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb>" %}
{% endcolumn %}

{% column %}
**Llama-3.2-1B-FP8 RL Colab**

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama_FP8_GRPO.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

### :cd:Implementing FP8 Training

Our first reference point was `transformers`, which already supports FP8 in a couple of ways. One of them is a block-quantized matmul implementation: when a layer receives 16‑bit activations, it quantizes them and passes them to a custom FP8 matmul kernel. After wiring this up and benchmarking on an NVIDIA H100, we saw the opposite of what we wanted: fine-tuning became about **4× slower** than standard BF16 fine-tuning.

### 🔥TorchAO Collab

So we worked with the [TorchAO](https://github.com/pytorch/ao) team (huge thanks to[ Andrew](https://github.com/unslothai/unsloth/pull/3440)) to incorporate TorchAO’s FP8 support into our RL workloads and saw around **1.4× faster throughput** and up to **60% less model memory usage**. At a high level:

* We store the frozen LoRA weights in FP8.
* During the forward pass, we apply dynamic FP8 quantization to the input activations, while keeping the trainable LoRA adapters in BF16.
* These FP8 weights share the same buffers as the vLLM model weights, so there’s only a single FP8 copy of the model in memory at any time (no “double model” memory overhead).
* In the backward pass, we dequantize the LoRA weights so all gradient computation is done in BF16 for better accuracy.

This general setup works across all supported RL algorithms, including [GSPO](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation/gspo-reinforcement-learning), Dr. GRPO, PPO, and DPO.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FUir0hB7T0xBtWUTnK3aG%2Funknown.png?alt=media&amp;token=d225cd2e-fdf4-4521-8e9f-72bd684eb9e4" alt="" width="375"><figcaption></figcaption></figure>

TorchAO provides PyTorch-native FP8 support for both training and inference, offering a variety of scaling granularities including tensorwise, row-wise, and 128x128 blockwise (prototype). TorchAO’s FP8 support can improve inference throughput by up to [1.64x at 27B scale](https://huggingface.co/pytorch/gemma-3-27b-it-FP8/blob/main/README.md#results-h100-machine) with row-wise scaling granularity. For more details, visit the TorchAO [FP8 README](https://github.com/pytorch/ao/blob/main/torchao/float8/README.md).

#### TorchAO’s block-quantized FP8 matmul

We used TorchAO’s block‑quantized FP8 matmul implementation which provided:

* **80% of BF16 throughput**
* Without degrading loss or training stability

So for a while, this became our default FP8 matmul backend, until FBGEMM caught up - we know default to using FBGEMM's implementation, if your GPU supports it! The current version of Unsloth can automatically choose the best backend based on what’s installed. If you have the right packages, you don’t have to leave performance on the table 🙂

PS: We also experimented with DeepSeek’s DeepGEMM, but couldn’t get it fully integrated end‑to‑end to run clean, apples‑to‑apples comparisons.

### :bird:On the fly TorchAO FP8 quantization

Massive thanks to [Andrew](https://github.com/unslothai/unsloth/pull/3440) from TorchAO, Unsloth FP8 RL also lets you quantize the model on the fly by doing quantization within the model load time and passing that on to vLLM. This way, you need not explicitly quantize the model yourself (we handle it for you). You can do this by setting `load_in_fp8 = True` in the model load arguments, and will do offline FP8 if we don't find a suitable pre-quantized checkpoint.

```python
from unsloth import FastLanguageModel
fp8_model = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.3-70B-Instruct", # Can be any model name!
    load_in_fp8 = True, # Can be "block" for block FP8, True for row FP8, False
)
```

### :tada:Unsloth FP8 uploads

For convenience, we uploaded FP8 Dynamic and FP8 Block models on Hugging Face. You can use them for FP8 training or also efficient & fast serving/deployment via [vLLM](/docs/basics/inference-and-deployment/vllm-guide)/[SGLang](/docs/basics/inference-and-deployment/sglang-guide) etc.

FP8 Dynamic offers slightly faster training and lower VRAM usage than FP8 Block, but with a small trade-off in accuracy. [See here](/docs/get-started/unsloth-model-catalog#fp8) for our full list of FP8 quants, but here the most popular ones:

| Model                 | FP8 uploads                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Qwen3 (2507)**      | <p>4B Instruct — <a href="https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-FP8">FP8</a><br>4B Thinking — <a href="https://huggingface.co/unsloth/Qwen3-4B-Thinking-2507-FP8">FP8</a><br>30B-A3B Instruct — <a href="https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-FP8">FP8</a><br>30B-A3B Thinking — <a href="https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-FP8">FP8</a></p>                                                                                                                                                                                                                                                                                                                                 |
| **Qwen3-VL**          | <p>4B Instruct — <a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-FP8">FP8</a><br>4B Thinking — <a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-FP8">FP8</a><br>8B Instruct — <a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-FP8">FP8</a><br>8B Thinking — <a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-FP8">FP8</a></p>                                                                                                                                                                                                                                                                                                                                                             |
| **Llama 3.1**         | <p>8B Instruct — <a href="https://huggingface.co/unsloth/Llama-3.1-8B-Instruct-FP8-Dynamic">Dynamic</a> · <a href="https://huggingface.co/unsloth/Llama-3.1-8B-Instruct-FP8-Block">Block</a><br>8B Base — <a href="https://huggingface.co/unsloth/Llama-3.1-8B-FP8-Dynamic">Dynamic</a> · <a href="https://huggingface.co/unsloth/Llama-3.1-8B-FP8-Block">Block</a><br>70B — <a href="https://huggingface.co/unsloth/Llama-3.1-70B-FP8-Dynamic">Dynamic</a> · <a href="https://huggingface.co/unsloth/Llama-3.1-70B-FP8-Block">Block</a></p>                                                                                                                                                                                                |
| **Qwen3**             | <p>0.6B — <a href="https://huggingface.co/unsloth/Qwen3-0.6B-FP8">FP8</a><br>1.7B — <a href="https://huggingface.co/unsloth/Qwen3-1.7B-FP8">FP8</a><br>4B — <a href="https://huggingface.co/unsloth/Qwen3-4B-FP8">FP8</a><br>8B — <a href="https://huggingface.co/unsloth/Qwen3-8B-FP8">FP8</a><br>14B — <a href="https://huggingface.co/unsloth/Qwen3-14B-FP8">FP8</a><br>32B — <a href="https://huggingface.co/unsloth/Qwen3-32B-FP8">FP8</a></p>                                                                                                                                                                                                                                                                                         |
| **Llama 3.3**         | 70B — [Dynamic](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-FP8-Dynamic) · [Block](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-FP8-Block)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| **Llama 3.2**         | <p>1B Base — <a href="https://huggingface.co/unsloth/Llama-3.2-1B-FP8-Dynamic">Dynamic</a> · <a href="https://huggingface.co/unsloth/Llama-3.2-1B-FP8-Block">Block</a><br>1B Instruct — <a href="https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-FP8-Dynamic">Dynamic</a> · <a href="https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-FP8-Block">Block</a><br>3B Base — <a href="https://huggingface.co/unsloth/Llama-3.2-3B-FP8-Dynamic">Dynamic</a> · <a href="https://huggingface.co/unsloth/Llama-3.2-3B-FP8-Block">Block</a><br>3B Instruct — <a href="https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-FP8-Dynamic">Dynamic</a> · <a href="https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-FP8-Block">Block</a></p> |
| **Granite 4.0**       | <p>h-tiny — <a href="https://huggingface.co/unsloth/granite-4.0-h-tiny-FP8-Dynamic">FP8 Dynamic</a><br>h-small — <a href="https://huggingface.co/unsloth/granite-4.0-h-small-FP8-Dynamic">FP8 Dynamic</a></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| **Magistral Small**   | [FP8 Dynamic](https://huggingface.co/unsloth/Magistral-Small-2509-FP8-Dynamic) · [FP8 torchao](https://huggingface.co/unsloth/Magistral-Small-2509-FP8-torchao)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| **Mistral Small 3.2** | [FP8](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-FP8)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| **Gemma 3**           | <p>270m — <a href="https://huggingface.co/unsloth/gemma-3-270m-it-FP8-Dynamic">FP8</a><br>1B — <a href="https://huggingface.co/unsloth/gemma-3-1b-it-FP8-Dynamic">FP8</a><br>4B — <a href="https://huggingface.co/unsloth/gemma-3-4b-it-FP8-Dynamic">FP8</a><br>12B — <a href="https://huggingface.co/unsloth/gemma-3-12B-it-FP8-Dynamic">FP8</a><br>27B — <a href="https://huggingface.co/unsloth/gemma-3-27b-it-FP8-Dynamic">FP8</a></p>                                                                                                                                                                                                                                                                                                  |

### :person\_tipping\_hand:Acknowledgements

Huge thanks to the entire PyTorch and TorchAO team for their help and collaboration! A huge thank you especially to: Andrew Or, Jerry Zhang, Supriya Rao, Scott Roy and Mergen Nachin for helping on many discussions on FP8 RL, and on helping to integrate it into Unsloth! Also thanks to the Executorch team as well!


# Tutorial: Train your own Reasoning model with GRPO

Beginner's Guide to transforming a model like Llama 3.1 (8B) into a reasoning model by using Unsloth and GRPO.

DeepSeek developed [GRPO](https://unsloth.ai/blog/grpo) (Group Relative Policy Optimization) to train their R1 reasoning models.

### Quickstart

These instructions are for our pre-made Google Colab [notebooks](/docs/get-started/unsloth-notebooks). If you are installing Unsloth locally, you can also copy our notebooks inside your favorite code editor. We'll be using any of these notebooks:

| [**Qwen3.5 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)_Vision_GRPO.ipynb) **- Vision - new** | [**gpt-oss-20b**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) **-** GSPO       | [Gemma 3 (4B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision-GRPO.ipynb) - Vision GSPO         |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| [**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-GRPO.ipynb) - Advanced                    | [Qwen3-VL-8B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb) - Vision GSPO | [Llama 3.2 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_\(3B\)_GRPO_LoRA.ipynb) - Advanced |

{% stepper %}
{% step %}

#### Install Unsloth

If you're using our Colab notebook, click **Runtime > Run all**. We'd highly recommend you checking out our [Fine-tuning Guide](/docs/get-started/fine-tuning-llms-guide) before getting started.

If installing locally, ensure you have the correct [requirements](/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) and use `pip install unsloth` on Linux or follow our [Windows install ](/docs/get-started/install/windows-installation)instructions.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-313fa39c229225ae9d39b7c7a0d05c9005ddb94c%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Learn about GRPO & Reward Functions

Before we get started, it is recommended to learn more about GRPO, reward functions and how they work. Read more about them including [tips & tricks](/docs/get-started/reinforcement-learning-rl-guide#basics-tips)[ here](/docs/get-started/reinforcement-learning-rl-guide#basics-tips).

You will also need enough VRAM. In general, model parameters = amount of VRAM you will need. In Colab, we are using their free 16GB VRAM GPUs which can train any model up to 16B in parameters.
{% endstep %}

{% step %}

#### Configure desired settings

We have pre-selected optimal settings for the best results for you already and you can change the model to whichever you want listed in our [supported models](/docs/get-started/unsloth-model-catalog). Would not recommend changing other settings if you're a beginner.

{% hint style="success" %}
For **advanced GRPO** documentation on batching, generation and training parameters, [read our guide!](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation)
{% endhint %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-b1e9fac448706ac87dff7e7eff1298655dda456e%2Fimage.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Data preparation

We have pre-selected OpenAI's [GSM8K](https://huggingface.co/datasets/openai/gsm8k) dataset which contains grade school math problems but you could change it to your own or any public one on Hugging Face. You can read more about [datasets here](/docs/get-started/fine-tuning-llms-guide/datasets-guide).

Your dataset should still have at least 2 columns for question and answer pairs. However the answer must not reveal the reasoning behind how it derived the answer from the question. See below for an example:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-14a1ee796547f725abbd1097f2b0f9e4e6cc5976%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

We'll structure the data to prompt the model to articulate its reasoning before delivering an answer. To start, we'll establish a clear format for both prompts and responses.

```
# Define the system prompt that instructs the model to use a specific format
SYSTEM_PROMPT = """
Respond in the following format:
<reasoning>
...
</reasoning>
<answer>
...
</answer>
"""

XML_COT_FORMAT = """\
<reasoning>
{reasoning}
</reasoning>
<answer>
{answer}
</answer>
"""
```

Now, to prepare the dataset:

```
import re
from datasets import load_dataset, Dataset


# Helper functions to extract answers from different formats
def extract_xml_answer(text: str) -> str:
    answer = text.split("<answer>")[-1]
    answer = answer.split("</answer>")[0]
    return answer.strip()


def extract_hash_answer(text: str) -> str | None:
    if "####" not in text:
        return None
    return text.split("####")[1].strip()


# Function to prepare the GSM8K dataset
def get_gsm8k_questions(split="train") -> Dataset:
    data = load_dataset("openai/gsm8k", "main")[split]
    data = data.map(
        lambda x: {
            "prompt": [
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": x["question"]},
            ],
            "answer": extract_hash_answer(x["answer"]),
        }
    )
    return data


dataset = get_gsm8k_questions()
```

The dataset is prepared by extracting the answers and formatting them as structured strings.
{% endstep %}

{% step %}

#### Reward Functions/Verifier

[Reward Functions/Verifiers](/docs/get-started/reinforcement-learning-rl-guide#reward-functions-verifier) lets us know if the model is doing well or not according to the dataset you have provided. Each generation run will be assessed on how it performs to the score of the average of the rest of generations. You can create your own reward functions however we have already pre-selected them for you with [Will's GSM8K](/docs/get-started/reinforcement-learning-rl-guide#gsm8k-reward-functions) reward functions. With this, we have 5 different ways which we can reward each generation.

You can input your generations into an LLM like ChatGPT 4o or Llama 3.1 (8B) and design a reward function and verifier to evaluate it. For example, feed your generations into a LLM of your choice and set a rule: "If the answer sounds too robotic, deduct 3 points." This helps refine outputs based on quality criteria. **See examples** of what they can look like [here](/docs/get-started/reinforcement-learning-rl-guide#reward-function-examples).

**Example Reward Function for an Email Automation Task:**

* **Question:** Inbound email
* **Answer:** Outbound email
* **Reward Functions:**
  * If the answer contains a required keyword → **+1**
  * If the answer exactly matches the ideal response → **+1**
  * If the response is too long → **-1**
  * If the recipient's name is included → **+1**
  * If a signature block (phone, email, address) is present → **+1**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-95cd00b6a52b8161b31a2399e25863ee0349920e%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Train your model

We have pre-selected hyperparameters for the most optimal results however you could change them. Read all about [parameters here](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide). For **advanced GRPO** documentation on batching, generation and training parameters, [read our guide!](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation)

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a22d3475d925d2d858c9fcc228f0e13893eff0f9%2Fimage.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

The **GRPOConfig** defines key hyperparameters for training:

* `use_vllm`: Activates fast inference using vLLM.
* `learning_rate`: Determines the model's learning speed.
* `num_generations`: Specifies the number of completions generated per prompt.
* `max_steps`: Sets the total number of training steps.

{% hint style="success" %}
**NEW!** We now support DAPO, Dr. GRPO and most other new GRPO techniques. You can play with the following arguments in GRPOConfig to enable:

```python
epsilon=0.2,
epsilon_high=0.28, # one sided
delta=1.5 # two sided

loss_type='bnpo',
# or:
loss_type='grpo',
# or:
loss_type='dr_grpo',
# or:
loss_type='dapo',

mask_truncated_completions=True,
```

{% endhint %}

You should see the reward increase overtime. We would recommend you train for at least 300 steps which may take 30 mins however, for optimal results, you should train for longer.

{% hint style="warning" %}
If you're having issues with your GRPO model not learning, we'd highly recommend to use our [Advanced GRPO notebooks](/docs/get-started/unsloth-notebooks#grpo-reasoning-notebooks) as it has a much better reward function and you should see results much faster and frequently.
{% endhint %}

You will also see sample answers which allows you to see how the model is learning. Some may have steps, XML tags, attempts etc. and the idea is as trains it's going to get better and better because it's going to get scored higher and higher until we get the outputs we desire with long reasoning chains of answers.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-f33d6f494605ab9ca69a0b697ed5865dd3a30b18%2Fimage.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run & Evaluate your model

Run your model by clicking the play button. In the first example, there is usually no reasoning in the answer and in order to see the reasoning, we need to first save the LoRA weights we just trained with GRPO first using:

<pre><code><strong>model.save_lora("grpo_saved_lora")
</strong></code></pre>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1ab351622655983aeda4d9d6d217cf354cb280be%2Fimage%20(10)%20(1)%20(1).png?alt=media" alt=""><figcaption><p>The first inference example run has no reasoning. You must load the LoRA and test it to reveal the reasoning.</p></figcaption></figure>

Then we load the LoRA and test it. Our reasoning model is much better - it's not always correct, since we only trained it for an hour or so - it'll be better if we extend the sequence length and train for longer!

You can then save your model to GGUF, Ollama etc. by following our [guide here](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/pages/nw2c1elNySGBBav8WP9B#id-7.-running--saving-the-model).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-38fa0c97184487aaa6b259f5b23b7f27345871d8%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

If you are still not getting any reasoning, you may have either trained for too less steps or your reward function/verifier was not optimal.
{% endstep %}

{% step %}

#### Save your model

We have multiple options for saving your fine-tuned model, but we’ll focus on the easiest and most popular approaches which you can read more about [here](/docs/basics/inference-and-deployment)

**Saving in 16-bit Precision**

You can save the model with 16-bit precision using the following command:

```python
# Save to 16-bit precision
model.save_pretrained_merged("model", tokenizer, save_method="merged_16bit")
```

**Pushing to Hugging Face Hub**

To share your model, we’ll push it to the Hugging Face Hub using the `push_to_hub_merged` method. This allows saving the model in multiple quantization formats.

```python
# Push to Hugging Face Hub (requires a token)
model.push_to_hub_merged(
    "your-username/model-name", tokenizer, save_method="merged_16bit", token="your-token"
)
```

**Saving in GGUF Format for llama.cpp**

Unsloth also supports saving in **GGUF format**, making it compatible with **llama.cpp** and **Ollama**.

```python
model.push_to_hub_gguf(
    "your-username/model-name",
    tokenizer,
    quantization_method=["q4_k_m", "q8_0", "q5_k_m"],
    token="your-token",
)
```

Once saved in GGUF format, the model can be easily deployed in lightweight environments using **llama.cpp** or used in other inference engines.
{% endstep %}
{% endstepper %}

## Video Tutorials

Here are some video tutorials created by amazing YouTubers who we think are fantastic!

{% embed url="<https://www.youtube.com/watch?v=9t-BAjzBWj8>" %}

{% columns %}
{% column width="50%" %}
{% embed url="<https://www.youtube.com/watch?t=3289s&v=bbFEYPx9Hpo>" %}
Great to learn about how to prep your dataset and explanations behind Reinforcement Learning + GRPO basics
{% endembed %}

{% embed url="<https://www.youtube.com/watch?v=oF0_eMhzRaQ>" %}
{% endcolumn %}

{% column width="50%" %}
{% embed url="<https://www.youtube.com/watch?v=juOh1afy-IE>" %}

{% embed url="<https://www.youtube.com/watch?v=SoPE1cUz3Hs>" %}
Local GRPO on your own device
{% endembed %}
{% endcolumn %}
{% endcolumns %}


# Advanced Reinforcement Learning Documentation

Advanced documentation settings when using Unsloth with GRPO.

Detailed guides on doing GRPO with Unsloth for Batching, Generation & Training Parameters:

## Training Parameters

* **`beta`** *(float, default 0.0)*: KL coefficient.
  * `0.0` ⇒ no reference model loaded (lower memory, faster).
  * Higher `beta` constrains the policy to stay closer to the ref policy.
* **`num_iterations`** *(int, default 1)*: PPO epochs per batch (μ in the algorithm).\
  Replays data within each gradient accumulation step; e.g., `2` = two forward passes per accumulation step.
* **`epsilon`** *(float, default 0.2)*: Clipping value for token-level log-prob ratios (typical ratio range ≈ \[-1.2, 1.2] with default ε).
* **`delta`** *(float, optional)*: Enables **upper** clipping bound for **two-sided GRPO** when set. If `None`, standard GRPO clipping is used. Recommended `> 1 + ε` when enabled (per INTELLECT-2 report).
* **`epsilon_high`** *(float, optional)*: Upper-bound epsilon; defaults to `epsilon` if unset. DAPO recommends **0.28**.
* **`importance_sampling_level`** *(“token” | “sequence”, default "token")*:
  * `"token"`: raw per-token ratios (one weight per token).
  * `"sequence"`: average per-token ratios to a single sequence-level ratio.\
    GSPO shows sequence-level sampling often gives more stable training for sequence-level rewards.
* **`reward_weights`** *(list\[float], optional)*: One weight per reward. If `None`, all weights = 1.0.
* **`scale_rewards`** *(str|bool, default "group")*:
  * `True` or `"group"`: scale by **std within each group** (unit variance in group).
  * `"batch"`: scale by **std across the entire batch** (per PPO-Lite).
  * `False` or `"none"`: **no scaling**. Dr. GRPO recommends not scaling to avoid difficulty bias from std scaling.
* **`loss_type`** *(str, default "dapo")*:
  * `"grpo"`: normalizes over sequence length (length bias; not recommended).
  * `"dr_grpo"`: normalizes by a **global constant** (introduced in Dr. GRPO; removes length bias). Constant ≈ `max_completion_length`.
  * `"dapo"` **(default)**: normalizes by **active tokens in the global accumulated batch** (introduced in DAPO; removes length bias).
  * `"bnpo"`: normalizes by **active tokens in the local batch** only (results can vary with local batch size; equals GRPO when `per_device_train_batch_size == 1`).
* **`mask_truncated_completions`** *(bool, default False)*:\
  When `True`, truncated completions are excluded from loss (recommended by DAPO for stability).\
  **Note**: There are some KL issues with this flag, so we recommend to disable it.

  ```python
  # If mask_truncated_completions is enabled, zero out truncated completions in completion_mask
  if self.mask_truncated_completions:
      truncated_completions = ~is_eos.any(dim=1)
      completion_mask = completion_mask * (~truncated_completions).unsqueeze(1).int()
  ```

  This can zero out all `completion_mask` entries when many completions are truncated, making `n_mask_per_reward = 0` and causing KL to become NaN. [See](https://github.com/unslothai/unsloth-zoo/blob/e705f7cb50aa3470a0b6e36052c61b7486a39133/unsloth_zoo/rl_replacements.py#L184)
* **`vllm_importance_sampling_correction`** *(bool, default True)*:\
  Applies **Truncated Importance Sampling (TIS)** to correct off-policy effects when generation (e.g., vLLM / fast\_inference) differs from training backend.\
  In Unsloth, this is **auto-set to True** if you’re using vLLM/fast\_inference; otherwise **False**.
* **`vllm_importance_sampling_cap`** *(float, default 2.0)*:\
  Truncation parameter **C** for TIS; sets an upper bound on the importance sampling ratio to improve stability.
* **`dtype`** when choosing float16 or bfloat16, see [FP16 vs BF16 for RL](/docs/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation/fp16-vs-bf16-for-rl)

### RL on unsupported models:

You can also run RL with Unsloth on models that are not supported by vLLM, such as [Qwen3.5](/docs/models/qwen3.5/fine-tune). Simply set `fast_inference=False` when loading the model.

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3.5-4B",
    fast_inference=False,
)
```

## Generation Parameters

* `temperature (float, defaults to 1.0):`\
  Temperature for sampling. The higher the temperature, the more random the completions. Make sure you use a relatively high (1.0) temperature to have diversity in generations which helps learning.
* `top_p (float, optional, defaults to 1.0):`\
  Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1.0 to consider all tokens.
* `top_k (int, optional):`\
  Number of highest probability vocabulary tokens to keep for top-k-filtering. If None, top-k-filtering is disabled and all tokens are considered.
* `min_p (float, optional):`\
  Minimum token probability, which will be scaled by the probability of the most likely token. It must be a value between 0.0 and 1.0. Typical values are in the 0.01-0.2 range.
* `repetition_penalty (float, optional, defaults to 1.0):`\
  Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1.0 encourage the model to use new tokens, while values < 1.0 encourage the model to repeat tokens.
* `steps_per_generation: (int, optional):`\
  Number of steps per generation. If None, it defaults to `gradient_accumulation_steps`. Mutually exclusive with `generation_batch_size`.

{% hint style="info" %}
It is a bit confusing to mess with this parameter, it is recommended to edit `per_device_train_batch_size` and gradient accumulation for the batch sizes
{% endhint %}

## Batch & Throughput Parameters

### Parameters that control batches

* **`train_batch_size`**: Number of samples **per process** per step.\
  If this integer is **less than `num_generations`**, it will default to `num_generations`.
* **`steps_per_generation`**: Number of **microbatches** that contribute to **one generation’s** loss calculation (forward passes only).\
  A new batch of data is generated every `steps_per_generation` steps; backpropagation timing depends on `gradient_accumulation_steps`.
* **`num_processes`**: Number of distributed training processes (e.g., GPUs / workers).
* **`gradient_accumulation_steps`** (aka `gradient_accumulation`): Number of microbatches to accumulate **before** applying backpropagation and optimizer update.
* **Effective batch size**:

  ```
  effective_batch_size = steps_per_generation * num_processes * train_batch_size
  ```

  Total samples contributing to gradients before an update (across all processes and steps).
* **Optimizer steps per generation**:

  ```
  optimizer_steps_per_generation = steps_per_generation / gradient_accumulation_steps
  ```

  Example: `4 / 2 = 2`.
* **`num_generations`**: Number of generations produced **per prompt** (applied **after** computing `effective_batch_size`).\
  The number of **unique prompts** in a generation cycle is:

  ```
  unique_prompts = effective_batch_size / num_generations
  ```

  **Must be > 2** for GRPO to work.

### GRPO Batch Examples

The tables below illustrate how batches flow through steps, when optimizer updates occur, and how new batches are generated.

#### Example 1

```
num_gpus = 1
per_device_train_batch_size = 3
gradient_accumulation_steps = 2
steps_per_generation = 4

effective_batch_size = 4 * 3 * 1 = 12
num_generations = 3
```

**Generation cycle A**

| Step | Batch    | Notes                                  |
| ---: | -------- | -------------------------------------- |
|    0 | \[0,0,0] |                                        |
|    1 | \[1,1,1] | → optimizer update (accum = 2 reached) |
|    2 | \[2,2,2] |                                        |
|    3 | \[3,3,3] | optimizer update                       |

**Generation cycle B**

| Step | Batch    | Notes                                  |
| ---: | -------- | -------------------------------------- |
|    0 | \[4,4,4] |                                        |
|    1 | \[5,5,5] | → optimizer update (accum = 2 reached) |
|    2 | \[6,6,6] |                                        |
|    3 | \[7,7,7] | optimizer update                       |

#### Example 2

```
num_gpus = 1
per_device_train_batch_size = 3
steps_per_generation = gradient_accumulation_steps = 4

effective_batch_size = 4 * 3 * 1 = 12
num_generations = 3
```

**Generation cycle A**

| Step | Batch    | Notes                                |
| ---: | -------- | ------------------------------------ |
|    0 | \[0,0,0] |                                      |
|    1 | \[1,1,1] |                                      |
|    2 | \[2,2,2] |                                      |
|    3 | \[3,3,3] | optimizer update (accum = 4 reached) |

**Generation cycle B**

| Step | Batch    | Notes                                |
| ---: | -------- | ------------------------------------ |
|    0 | \[4,4,4] |                                      |
|    1 | \[5,5,5] |                                      |
|    2 | \[6,6,6] |                                      |
|    3 | \[7,7,7] | optimizer update (accum = 4 reached) |

#### Example 3

```
num_gpus = 1
per_device_train_batch_size = 3
steps_per_generation = gradient_accumulation_steps = 4

effective_batch_size = 4 * 3 * 1 = 12
num_generations = 4
unique_prompts = effective_batch_size / num_generations = 3
```

**Generation cycle A**

| Step | Batch    | Notes                                |
| ---: | -------- | ------------------------------------ |
|    0 | \[0,0,0] |                                      |
|    1 | \[0,1,1] |                                      |
|    2 | \[1,1,3] |                                      |
|    3 | \[3,3,3] | optimizer update (accum = 4 reached) |

**Generation cycle B**

| Step | Batch    | Notes                                |
| ---: | -------- | ------------------------------------ |
|    0 | \[4,4,4] |                                      |
|    1 | \[4,5,5] |                                      |
|    2 | \[5,5,6] |                                      |
|    3 | \[6,6,6] | optimizer update (accum = 4 reached) |

#### Example 4

```
num_gpus = 1
per_device_train_batch_size = 6
steps_per_generation = gradient_accumulation_steps = 2

effective_batch_size = 2 * 6 * 1 = 12
num_generations = 3
unique_prompts = 4
```

**Generation cycle A**

| Step | Batch           | Notes                                |
| ---: | --------------- | ------------------------------------ |
|    0 | \[0,0,0, 1,1,1] |                                      |
|    1 | \[2,2,2, 3,3,3] | optimizer update (accum = 2 reached) |

**Generation cycle B**

| Step | Batch           | Notes                                |
| ---: | --------------- | ------------------------------------ |
|    0 | \[4,4,4, 5,5,5] |                                      |
|    1 | \[6,6,6, 7,7,7] | optimizer update (accum = 2 reached) |

### Quick Formula Reference

```
effective_batch_size = steps_per_generation * num_processes * train_batch_size
optimizer_steps_per_generation = steps_per_generation / gradient_accumulation_steps
unique_prompts = effective_batch_size / num_generations   # must be > 2
```


# GSPO Reinforcement Learning

Train with GSPO (Group Sequence Policy Optimization) RL in Unsloth.

We're introducing GSPO which is a variant of [GRPO](/docs/get-started/reinforcement-learning-rl-guide#from-rlhf-ppo-to-grpo-and-rlvr) made by the Qwen team at Alibaba. They noticed the observation that when GRPO takes importance weights for each token, even though inherently advantages do not scale or change with each token. This lead to the creation of GSPO, which now assigns the importance on the sequence likelihood rather than the individual token likelihoods of the tokens.

* Use our free GSPO notebooks for: [**gpt-oss-20b**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) and [**Qwen2.5-VL**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb)

Enable GSPO in Unsloth by setting `importance_sampling_level = "sequence"` in the GRPO config. The difference between these two algorithms can be seen below, both from the GSPO paper from Qwen and Alibaba:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-45d743dd5dcd590626777ce09cfab61808aa8c24%2Fimage.png?alt=media" alt="" width="563"><figcaption><p>GRPO Algorithm, Source: <a href="https://arxiv.org/abs/2507.18071">Qwen</a></p></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-ee755850cbe17482ce240dde227d55c62e9a3e64%2Fimage.png?alt=media" alt="" width="563"><figcaption><p>GSPO algorithm, Source: <a href="https://arxiv.org/abs/2507.18071">Qwen</a></p></figcaption></figure>

In Equation 1, it can be seen that the advantages scale each of the rows into the token logprobs before that tensor is sumed. Essentially, each token is given the same scaling even though that scaling was given to the entire sequence rather than each individual token. A simple diagram of this can be seen below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-b3c944808a15dde0a7ff45782f9f074993304bf1%2FCopy%20of%20GSPO%20diagram%20(1).jpg?alt=media" alt="" width="286"><figcaption><p>GRPO Logprob Ratio row wise scaled with advantages</p></figcaption></figure>

Equation 2 shows that the logprob ratios for each sequence is summed and exponentiated after the Logprob ratios are computed, and only the resulting now sequence ratios get row wise multiplied by the advantages.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-62fc5b50921e79cce155d2794201c9b96faf941e%2FGSPO%20diagram%20(1).jpg?alt=media" alt="" width="313"><figcaption><p>GSPO Sequence Ratio row wise scaled with advantages</p></figcaption></figure>

Enabling GSPO is simple, all you need to do is set the `importance_sampling_level = "sequence"` flag in the GRPO config.

```python
training_args = GRPOConfig(
    output_dir = "vlm-grpo-unsloth",
    per_device_train_batch_size = 8,
    gradient_accumulation_steps = 4,
    learning_rate = 5e-6,
    adam_beta1 = 0.9,
    adam_beta2 = 0.99,
    weight_decay = 0.1,
    warmup_ratio = 0.1,
    lr_scheduler_type = "cosine",
    optim = "adamw_8bit",
    # beta = 0.00,
    epsilon = 3e-4,
    epsilon_high = 4e-4,
    num_generations = 8,    
    max_prompt_length = 1024,
    max_completion_length = 1024,
    log_completions = False,
    max_grad_norm = 0.1,
    temperature = 0.9,
    # report_to = "none", # Set to "wandb" if you want to log to Weights & Biases
    num_train_epochs = 2, # For a quick test run, increase for full training
    
    # GSPO is below:
    importance_sampling_level = "sequence",
    
    # Dr GRPO / GAPO etc
    loss_type = "dr_grpo",
)
```


# RL Reward Hacking

Learn what is Reward Hacking in Reinforcement Learning and how to counter it.

The ultimate goal of RL is to maximize some reward (say speed, revenue, some metric). But RL can **cheat.** When the RL algorithm learns a trick or exploits something to increase the reward, without actually doing the task at end, this is called "**Reward Hacking**".

It's the reason models learn to modify unit tests to pass coding challenges, and these are critical blockers for real world deployment. Some other good examples are from [Wikipedia](https://en.wikipedia.org/wiki/Reward_hacking).

<div align="center"><figure><img src="https://i.pinimg.com/originals/55/e0/1b/55e01b94a9c5546b61b59ae300811c83.gif" alt="" width="188"><figcaption></figcaption></figure></div>

**Can you counter reward hacking? Yes!** In our [free gpt-oss RL notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) we explore how to counter reward hacking in a code generation setting and showcase tangible solutions to common error modes. We saw the model edit the timing function, outsource to other libraries, cache the results, and outright cheat. After countering, the result is our model generates genuinely optimized matrix multiplication kernels, not clever cheats.

## :trophy: Reward Hacking Overview

Some common examples of reward hacking during RL include:

#### Laziness

RL learns to use Numpy, Torch, other libraries, which calls optimized CUDA kernels. We can stop the RL algorithm from calling optimized code by inspecting if the generated code imports other non standard Python libraries.

#### Caching & Cheating

RL learns to cache the result of the output and RL learns to find the actual output by inspecting Python global variables.

We can stop the RL algorithm from using cached data by wiping the cache with a large fake matrix. We also have to benchmark carefully with multiple loops and turns.

#### Cheating

RL learns to edit the timing function to make it output 0 time as passed. We can stop the RL algorithm from using global or cached variables by restricting it's `locals` and `globals`. We are also going to use `exec` to create the function, so we have to save the output to an empty dict. We also disallow global variable access via `types.FunctionType(f.__code__, {})`\\


# FP16 vs BF16 for RL

Defeating the Training-Inference Mismatch via FP16 https\://arxiv.org/pdf/2510.26788 shows how using float16 is better than bfloat16

### Float16 vs Bfloat16

There was a paper titled "**Defeating the Training-Inference Mismatch via FP16**" <https://arxiv.org/pdf/2510.26788> showing how using float16 precision can dramatically be better than using bfloat16 when doing reinforcement learning.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Frec4qe1aQS0xyMzGvS9c%2Fimage.png?alt=media&amp;token=2137e766-0f1f-48ec-b25f-2292d6f149f4" alt=""><figcaption></figcaption></figure>

In fact the longer the generation, the worse it gets when using bfloat16:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FWs7ioB2lraTbDbUCOAnn%2Fimage.png?alt=media&amp;token=ac2b4f8e-210f-4bcc-bcbb-6e68f80781a6" alt=""><figcaption></figcaption></figure>

We did an investigation, and **DO find float16 to be more stable** than bfloat16 with much smaller gradient norms see <https://x.com/danielhanchen/status/1985557028295827482> and <https://x.com/danielhanchen/status/1985562902531850472>

{% columns %}
{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhvQ1W5wtV6TTfsetp7y2%2FG44d7ZFbIAANBBd.jpg?alt=media&amp;token=35181a07-de3e-4321-b54e-4436b4a201ff" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F62HkxnGcaKvxnSxbZMZu%2FG44c20SbwAAGo8j.jpg?alt=media&amp;token=e0c7ecb8-6f0c-4ecf-b1a0-50f1b2a9a807" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fsi18IkGqE4IuUvzroyHh%2FG44ix5FbQAM0L5l.jpg?alt=media&amp;token=bc3b97ce-5df4-4b69-aa50-a8e339f21601" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### :exploding\_head:A100 Cascade Attention Bug

As per <https://x.com/RichardYRLi/status/1984858850143715759> and <https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda>, older vLLM versions (before 0.11.0) had broken attention mechanisms for A100 and similar GPUs. Please update vLLM! We also by default disable cascade attention in vLLM during Unsloth reinforcement learning if we detect an older vLLM version.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FnkCLRVIIGLADXBSCe58e%2Fimage.png?alt=media&amp;token=6669642f-8690-44bf-b2de-6aa89acf2332" alt=""><figcaption></figcaption></figure>

Different hardware also changes results, where newer and more expensive GPUs have less KL difference between the inference and training sides:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FaroTTz68zzyofy6nagtH%2Fimage.webp?alt=media&amp;token=3be09506-b8a0-42eb-8d17-af72496a9cd1" alt=""><figcaption></figcaption></figure>

### :fire:Using float16 in Unsloth RL

To use float16 precision in Unsloth GRPO and RL, you just need to set `dtype = torch.float16` and we'll take care of the rest!

{% code overflow="wrap" %}

```python
from unsloth import FastLanguageModel
import torch
max_seq_length = 2048 # Can increase for longer reasoning traces
lora_rank = 32 # Larger rank = smarter, but slower

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-4B-Base",
    max_seq_length = max_seq_length,
    load_in_4bit = False, # False for LoRA 16bit
    fast_inference = True, # Enable vLLM fast inference
    max_lora_rank = lora_rank,
    gpu_memory_utilization = 0.9, # Reduce if out of memory
    
    dtype = torch.float16, # Use torch.float16, torch.bfloat16
)
```

{% endcode %}


# Memory Efficient RL

We're excited to introduce more efficient reinforcement learning (RL) in Unsloth with multiple algorithmic advancements:

* **1.2 to 1.7x increased context lengths** with no slowdown and no extra memory usage!
* **10% faster RL training runs** with revamped kernels and async data movements
* **2x faster `torch.compile` times** during model loading

Unsloth **already** increases RL training speed, context window and reduces VRAM usage by 50–90% vs. all other setups with FA2, but now [**Unsloth's Standby**](#unsloth-standby) improves this even further. Our Standby feature uniquely limits speed degradation compared to other implementations and sometimes makes training even faster!

Now, Qwen3-32B LoRA 16-bit can attain 6,144 context lengths vs 3,600 (**1.7x longer**) before on 1xH100 80GB GPU. Llama-3.1-8B QLoRA 4bit can attain 47,500 lengths vs 42,000 before (1.13x longer).

We made RL runs 10% faster through various kernel optimizations, and removed the LoRA communication channel between the CPU and GPU when switching from training to inference mode. Finally, we used custom `torch.compile` flags to make vLLM's rollout faster by 10%, and reduced compilation time by 2x.

## :sparkles:How to enable optimizations

To enable **Unsloth's Standby** feature, set the environment variable `UNSLOTH_VLLM_STANDBY` before any Unsloth import. Then set `gpu_memory_utilization = 0.95` and that's it!

```python
import os
os.environ["UNSLOTH_VLLM_STANDBY"] = "1"

from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-8B-Base",
    max_seq_length = 2048, # Can increase for longer reasoning traces
    load_in_4bit = False, # False for LoRA 16bit
    fast_inference = True,
    max_lora_rank = 32, # Larger rank = smarter, but slower
    gpu_memory_utilization = 0.95,
)
```

## :mortar\_board:No more `gpu_memory_utilization`!

With Unsloth's new RL improvements, you NEVER have to worry about tuning or setting `gpu_memory_utilization` ever again - simply set it to 90% or 95% of GPU utilization - 100% sadly won't work since some space is needed for small tensors. Previously one had to tune it from 30% to 95% - no more now! Set it to the maximum and Unsloth will handle the rest!

## :interrobang:Why does RL use so much memory?

GRPO (and many RL variants) rely heavily on generation which is primarily powered by vLLM. But this comes comes with a steep cost since it requires constant **GPU memory for weights, activations, and the KV Cache**.

{% columns %}
{% column %}
Inference takes a lot of VRAM

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-7e25501083081b201d59f6000219cafa535d2b2d%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
Whilst Training also uses VRAM!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-189fd45a9e7a6fa1e98d1c9646b57bd0ec48481d%2Ffig6-2.avif?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

This means RL needs to keep 2 sets of VRAM / memory on the GPU at the same time:

1. Inference engine (has model weights, KV cache)
2. Training engine (has model weights, activations, gradients, optimizer states)

Current RL frameworks have to split 50/50 for a 80GB GPU with 50% for inference and 50% for training. And moving weights from training mode to inference mode can take quite some time.

<table><thead><tr><th width="251.51666259765625">80GB GPU</th><th>Inference Engine (50%)</th><th>Training Engine (50%)</th></tr></thead><tbody><tr><td>Model Weights</td><td>16GB</td><td>16GB</td></tr><tr><td>KV Cache</td><td>24GB</td><td></td></tr><tr><td>Activations, Gradients, Optimizer States</td><td></td><td>24GB</td></tr></tbody></table>

Previous Unsloth versions already smartly optimizes the above, as we **share vLLM's weight space directly which removes the double memory usage of the model weights**. This frees up 16GB of space for example which can be used to increase context length or the speed of generation. Also, we don't need to do memory movements, which makes training faster.

| 80GB GPU                                 | Inference Engine (50%) | Training Engine (50%) |
| ---------------------------------------- | ---------------------- | --------------------- |
| Model Weights                            | **16GB SHARED**        | **<<< SHARED**        |
| KV Cache                                 | 24GB + 8GB= **32GB**   |                       |
| Activations, Gradients, Optimizer States |                        | 24GB + 8GB=**32GB**   |

## 🦥Unsloth Standby

But we can go further - we first note RL does inference then training then inference then training etc.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-6e9b6a2f7381de84ed6eeb0feedc566cd443acf3%2F5b957843-eb58-4778-8b90-f25767c51495.png?alt=media" alt=""><figcaption></figcaption></figure>

This means the memory space for inference and training can in theory be re-used, since inference and training are separate modes - this is where [vLLM's sleep mode feature](https://docs.vllm.ai/en/latest/features/sleep_mode.html#rlhf-weight-updates) comes in, which has 2 options:

1. `level = 1` copies weights to the CPU and deletes KV cache
2. `level = 2` deletes weights and deletes KV cache

But reminder in Unsloth we share vLLM's memory space for the weights - this means we need a new way to delete the KV cache, and ignore deletion of the weights, and we call this Unsloth Standby.

| 80GB GPU                                                                                                                                                            | Inference Engine | Training Engine                          |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ---------------------------------------- |
| Model Weights                                                                                                                                                       | **16GB SHARED**  | **<<< SHARED**                           |
| <p><mark style="background-color:purple;"><strong>Multi-purpose</strong></mark></p><p><mark style="background-color:purple;"><strong>64GB space</strong></mark></p> | KV Cache         | Activations, Gradients, Optimizer States |

To enable this, simply add the below to all RL / GRPO training runs before any Unsloth import:

```python
import os
os.environ["UNSLOTH_VLLM_STANDBY"] = "1"
```

## 🧪Performance Experiments

Here you will find out how we benchmarked memory usage and context length for GRPO. Note that we do **2 generations per prompt because for GRPO to work**, we need at least 2 generations for which to calculate the sample mean and variance. **Without 2 generations, the standard deviation of one sample is 0**. This causes the advantages which uses this: (reward - mean)/std **to be undefined**.

$$
Z=\frac{r\_i - \mu}{\sqrt{\frac{1}{n}\sum(r\_i-\mu)^2}} \\
Z\_{n=1}=\frac{r\_1 - \mu}{\sqrt{\frac{1}{1}\sum(r\_1-\mu)^2}}=\frac{0}{0}=\text{undefined}
$$

This means for GRPO specifically, a maximum context length of 6,144 for Qwen-3 32B is actually 6,144 multiplied by 2 generations ie 12,288 in length.

We provide experiments for Llama-3.1 8B on both LoRA (16bit) and QLoRA (4bit) below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2f83185e373186aa67bc2ce7d1814b2edb0f3ce6%2Foutput%20(10).png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

**If you notice any training time differences, it isn’t much**. In our apples to apples comparison we noticed <1% training time slowdowns or even speedups which can be attributed to margin of error.

We also theorize speedups are possible due to reduced memory pressure, so there might be less memory cleanup on the CUDA memory allocator side.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-db26f62f9080dba942add171880537c3f516f065%2Fgpu%20mem%20cofigure.png?alt=media" alt=""><figcaption></figcaption></figure>

In the above image, you see the difference between baseline and standby mode on a single T4 GPU for Qwen 3 4B. <mark style="background-color:green;">**We can stretch the vllm's**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`gpu_memory_utilisation`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to as high as 0.95 without worrying that it'd affect training**</mark>. This means you can fit higher context length sequences and more sequences can be processed. In the first case, for example, we have enough memory to fit and process 32K length sequences provided training allows where as previously, any inputs longer than 2K would potentially not fit in and end up causing OOMs (out of memory).

<table data-full-width="true"><thead><tr><th>Experiments</th><th>Config</th><th>Status</th><th>GPU Memory usage</th><th>Comments</th></tr></thead><tbody><tr><td><ol><li><a href="https://colab.research.google.com/drive/18CssBY5C0mStnLvu2Hlt4aFLoPugRG0K?usp=sharing">u0.95gen2ga1s Qwen3_(4B)-GRPO.ipynb</a></li></ol></td><td><p><code>standby True</code></p><p><code>vllm_gpu_util 0.95</code></p><p><code>num_gen 2</code></p><p><code>grad_acc_steps 2</code></p></td><td>Runs for 40 steps/ 40 minutes</td><td><p>14.5 GiB (set by vllm_gpu_util)</p><p><br></p></td><td>Enough to fit in 32K KVCache with chunk of 2-4K or say 16K KVCache + 16K chunks</td></tr><tr><td><ol start="2"><li><a href="https://colab.research.google.com/drive/1q0TOUychygfreI2wKpg51sqnRhs5cYnX?usp=sharing">u9ge2ga2s Qwen3_(4B)-GRPO.ipynb</a></li></ol></td><td><p><code>standby True</code></p><p><code>vllm_gpu_util 0.9</code></p><p><code>num_gen 2</code></p><p><code>grad_acc_steps 2</code></p></td><td>Runs 32 steps in 40 m</td><td>13.8 GiB (set by…)</td><td>Approx enough to fit in ~28K KVCache with chunk of 2-4K or say 15K KVCache + 15K chunks</td></tr><tr><td><ol start="3"><li><a href="https://colab.research.google.com/drive/12Uw8y5beLzPtx11mCWCYyh9Z_PEHHdId?usp=sharing">u9ge2ga2ns Qwen3_(4B)-GRPO.ipynb</a></li></ol></td><td><p><code>standby False</code></p><p><code>vllm_gpu_util 0.9</code></p><p><code>num_gen 2</code></p><p><code>grad_acc_steps 2</code></p></td><td>model loads but can’t train because even batch size of 1 doesn’t fit</td><td>OOM</td><td><br></td></tr><tr><td><ol start="4"><li><a href="https://colab.research.google.com/drive/1GwTlaP5CLsW-BcE1LqZWkz6S8VTWYdJ2?usp=sharing">u8ge2ga2ns Qwen3_(4B)-GRPO.ipynb</a></li></ol></td><td><p><code>standby False</code></p><p><code>vllm_gpu_util 0.8</code></p><p><code>num_gen 2</code></p><p><code>grad_acc_steps 2</code></p></td><td>model loads but can’t train because even batch size of 1 doesn’t fit</td><td>OOM</td><td><br></td></tr><tr><td><ol start="5"><li><a href="https://colab.research.google.com/drive/1IuSUNzEBTiURK-vbTQuRDuUl0Ya2pz2t?usp=sharing">u7ge2ga2ns Qwen3_(4B)-GRPO.ipynb</a></li></ol></td><td><p><code>standby False</code></p><p><code>vllm_gpu_util 0.7</code></p><p><code>num_gen 2</code></p><p><code>grad_acc_steps 2</code></p></td><td><p>Trains fine</p><p>28 steps take 39min</p></td><td>~15.1GiB</td><td>any input slightly longer will result in OOM on colab</td></tr><tr><td><ol start="6"><li><a href="https://colab.research.google.com/drive/1RY7HwpZ0luJT70OyLJ6zXKZQ2COdT9QJ?usp=sharing">u7gen2ga2s Qwen3_(4B)-GRPO.ipynb</a></li></ol></td><td><p><code>standby True</code></p><p><code>vllm_gpu_util 0.7</code></p><p><code>num_gen 2</code></p><p><code>grad_acc_steps 2</code></p></td><td><p>Trains fine</p><p>29 steps take 40min</p></td><td>13GiB but most of the time around 10-11GB</td><td>At the same config, we save 2GiB aka 15% memory here.<br>Can be higher for longer sequences</td></tr></tbody></table>

### H100 Experiments

| Model                | GPU                   | Seq Len | Num Generations | Grad Acc Steps |
| -------------------- | --------------------- | ------- | --------------- | -------------- |
| Qwen2.5-14B-Instruct | NVIDIA H100 80GB PCIe | 32,768  | 8               | 4              |

In our collapsible results below, you can see there is a 9GiB difference in the peak memory used (note that 90% of the time, the GPU memory usage is equal to the peak memory in our case). **To put things into perspective, using TRL and LoRA we were able to only fine-tune an 8B parameter model with a context length of 1024 at max (32x less).** Anything with higher sequence length (with similar configuration) results in the process failing with OOM.

<details>

<summary>Click for Unsloth Standby Mode vs. no Standby Benchmarks</summary>

```
Standy mode enabled:

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |  32249 MiB |  43042 MiB | 128336 GiB | 128305 GiB |
|       from large pool |  31415 MiB |  42165 MiB | 127204 GiB | 127173 GiB |
|       from small pool |    834 MiB |   1184 MiB |   1132 GiB |   1131 GiB |
|---------------------------------------------------------------------------|
| Active memory         |  32249 MiB |  43042 MiB | 128336 GiB | 128305 GiB |
|       from large pool |  31415 MiB |  42165 MiB | 127204 GiB | 127173 GiB |
|       from small pool |    834 MiB |   1184 MiB |   1132 GiB |   1131 GiB |
|---------------------------------------------------------------------------|
| Requested memory      |  32199 MiB |  42987 MiB | 128176 GiB | 128145 GiB |
|       from large pool |  31364 MiB |  42110 MiB | 127047 GiB | 127016 GiB |
|       from small pool |    834 MiB |   1184 MiB |   1129 GiB |   1128 GiB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |  37644 MiB |  47504 MiB | 705806 MiB | 668162 MiB |
|       from large pool |  36376 MiB |  46588 MiB | 682818 MiB | 646442 MiB |
|       from small pool |   1268 MiB |   1284 MiB |  22988 MiB |  21720 MiB |
|---------------------------------------------------------------------------|
| Non-releasable memory | 713142 KiB |   4633 MiB | 103206 GiB | 103205 GiB |
|       from large pool | 525312 KiB |   4594 MiB | 101923 GiB | 101922 GiB |
|       from small pool | 187830 KiB |    250 MiB |   1283 GiB |   1283 GiB |
|---------------------------------------------------------------------------|
| Allocations           |    3460    |    4809    |   15606 K  |   15603 K  |
|       from large pool |     395    |     563    |    2812 K  |    2811 K  |
|       from small pool |    3065    |    4270    |   12794 K  |   12791 K  |
|---------------------------------------------------------------------------|
| Active allocs         |    3460    |    4809    |   15606 K  |   15603 K  |
|       from large pool |     395    |     563    |    2812 K  |    2811 K  |
|       from small pool |    3065    |    4270    |   12794 K  |   12791 K  |
|---------------------------------------------------------------------------|
| GPU reserved segments |     913    |     920    |   13260    |   12347    |
|       from large pool |     279    |     305    |    1766    |    1487    |
|       from small pool |     634    |     642    |   11494    |   10860    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |     422    |     628    |    4766 K  |    4765 K  |
|       from large pool |      66    |      92    |    1290 K  |    1289 K  |
|       from small pool |     356    |     555    |    3476 K  |    3475 K  |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|


Without Standby:

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |  32711 MiB |  52084 MiB | 142756 GiB | 142724 GiB |
|       from large pool |  31877 MiB |  51207 MiB | 141499 GiB | 141467 GiB |
|       from small pool |    834 MiB |   1184 MiB |   1257 GiB |   1256 GiB |
|---------------------------------------------------------------------------|
| Active memory         |  32711 MiB |  52084 MiB | 142756 GiB | 142724 GiB |
|       from large pool |  31877 MiB |  51207 MiB | 141499 GiB | 141467 GiB |
|       from small pool |    834 MiB |   1184 MiB |   1257 GiB |   1256 GiB |
|---------------------------------------------------------------------------|
| Requested memory      |  32572 MiB |  51658 MiB | 141898 GiB | 141866 GiB |
|       from large pool |  31738 MiB |  50780 MiB | 140644 GiB | 140613 GiB |
|       from small pool |    833 MiB |   1184 MiB |   1253 GiB |   1252 GiB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |  49552 MiB |  52188 MiB |  86354 MiB |  36802 MiB |
|       from large pool |  48320 MiB |  51300 MiB |  84740 MiB |  36420 MiB |
|       from small pool |   1232 MiB |   1232 MiB |   1614 MiB |    382 MiB |
|---------------------------------------------------------------------------|
| Non-releasable memory |      0 B   |      0 B   |      0 B   |      0 B   |
|       from large pool |      0 B   |      0 B   |      0 B   |      0 B   |
|       from small pool |      0 B   |      0 B   |      0 B   |      0 B   |
|---------------------------------------------------------------------------|
| Allocations           |    3460    |    4809    |   17440 K  |   17437 K  |
|       from large pool |     395    |     564    |    2742 K  |    2741 K  |
|       from small pool |    3065    |    4270    |   14698 K  |   14695 K  |
|---------------------------------------------------------------------------|
| Active allocs         |    3460    |    4809    |   17440 K  |   17437 K  |
|       from large pool |     395    |     564    |    2742 K  |    2741 K  |
|       from small pool |    3065    |    4270    |   14698 K  |   14695 K  |
|---------------------------------------------------------------------------|
| GPU reserved segments |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|
```

</details>

The image below shows how standby compares against non standby training with Unsloth. It is averaged over 3 runs to make sure the metrics aren’t noisy. In fact, if you zoom in close enough, you’d see that enabling standby makes it faster as well, probably due to less memory pressure as discussed before.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2f285043ea8afa38d1082513e424662d8cd04b90%2Ftrainglobalstep.png?alt=media" alt=""><figcaption></figcaption></figure>

### Previous A100 40GB experiments

In our previous experiments on A100 40GB GPU with Qwen-2.5-3b-instruct and 8 generations per sample, we observed that without standby, the GRPO training (model loaded in 16bit, LoRA, only weights trainable), we could only fit 6K sequence lengths. With our standby feature, we were able to fit 10K and beyond! **For comparison TRL can only give you context lengths of up to 1K while holding the same batch size.**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-c7cd807b5d513b04f5f3a6219bfcea0fb12e442a%2Fqwen3%20gpu%20mem.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

## :tada:Other optimizations

We now select better compilation flags and reduce compile times by 50% or more. We also managed to dynamically patch any vLLM version to handle `gc.collect` better for backwards compatibility reasons, as inspired from this [vLLM pull request](https://github.com/vllm-project/vllm/pull/21146). This reduces compilation times from 2 minutes to under 40 seconds.

We also optimized `torch.compile` flags and tried turning on some flags - unfortunately `combo_kernels` and `multi_kernel` could not function correctly on vLLM 0.10 and Torch 2.8/2.9 nightly and `coordinate_descent_tuning` made autotuning all kernels dramatically slower. It used to compile in under a minute, but enabling it took over 13 minutes and more, with minimal performance gains.

## :books:GRPO Notebooks

All our GRPO notebooks have Unsloth Standby on by default and all optimizations! See <https://docs.unsloth.ai/get-started/unsloth-notebooks> for all our GRPO notebooks, or try the below:

* [**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-GRPO.ipynb) **-** Advanced GRPO LoRA
* [**DeepSeek-R1-0528-Qwen3 (8B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_\(8B\)_GRPO.ipynb) (for multilingual usecases)
* [Gemma 3 (1B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(1B\)-GRPO.ipynb)
* [Llama 3.2 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_2_\(3B\)_GRPO_LoRA.ipynb) - Advanced GRPO LoRA
* [Llama 3.1 (8B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_\(8B\)-GRPO.ipynb)
* [Phi-4 (14B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_\(14B\)-GRPO.ipynb)
* [Mistral v0.3 (7B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-GRPO.ipynb)
* [Qwen2.5 (3B)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_\(3B\)-GRPO.ipynb)


# Preference Optimization Training - DPO, ORPO & KTO

Learn about preference alignment fine-tuning with DPO, GRPO, ORPO or KTO via Unsloth, follow the steps below:

DPO (Direct Preference Optimization), ORPO (Odds Ratio Preference Optimization), PPO, KTO Reward Modelling all work with Unsloth.

We have Google Colab notebooks for reproducing GRPO, ORPO, DPO Zephyr, KTO and SimPO:

* [GRPO notebooks](/docs/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks)
* [ORPO notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-ORPO.ipynb)
* [DPO Zephyr notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Zephyr_\(7B\)-DPO.ipynb)
* [KTO notebook](https://colab.research.google.com/drive/1MRgGtLWuZX4ypSfGguFgC-IblTvO2ivM?usp=sharing)
* [SimPO notebook](https://colab.research.google.com/drive/1Hs5oQDovOay4mFA6Y9lQhVJ8TnbFLFh2?usp=sharing)

We're also in 🤗Hugging Face's official docs! We're on the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and the [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth).

## DPO Code

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0" # Optional set GPU device ID

from unsloth import FastLanguageModel, PatchDPOTrainer
from unsloth import is_bfloat16_supported
PatchDPOTrainer()
import torch
from trl import DPOTrainer, DPOConfig  # Changed from TrainingArguments

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/zephyr-sft-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = None,
    load_in_4bit = True,
)

# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
    model,
    r = 64,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 64,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 3407,
    max_seq_length = max_seq_length,
)

dpo_trainer = DPOTrainer(
    model = model,
    ref_model = None,
    args = DPOConfig( # Use DPOConfig
        per_device_train_batch_size = 4,
        gradient_accumulation_steps = 8,
        warmup_ratio = 0.1,
        num_train_epochs = 3,
        fp16 = not is_bfloat16_supported(),
        bf16 = is_bfloat16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        seed = 42,
        output_dir = "outputs",
    ),
    beta = 0.1,
    train_dataset = YOUR_DATASET_HERE,
    # eval_dataset = YOUR_DATASET_HERE,
    tokenizer = tokenizer,
    max_length = 1024,
    max_prompt_length = 512,
)

dpo_trainer.train()
```


# Training AI Agents with RL

Learn how to train AI agents for real-world tasks using Reinforcement Learning (RL).

“Agentic” AI is becoming more popular over time. In this context, an “agent” is an LLM that is given a high-level goal and a set of tools to achieve it. Agents are also typically “multi-turn” — they can perform an action, see what effect it had on the environment, and then perform another action repeatedly, until they achieve their goal or fail trying.

Unfortunately, even very capable LLMs can have a hard time performing complex multi-turn agentic tasks reliably. Interestingly, we’ve found that training agents using an RL algorithm called [GRPO (Group Relative Policy Optimization)](/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo) can make them far more reliable! In this guide, you will learn how to to build reliable AI agents using open-source tools.

## 🎨 Training RL Agents with ART

[ART (Agent Reinforcement Trainer)](https://github.com/openpipe/art) built on top of [Unsloth](https://github.com/unslothai/unsloth)’s GRPOTrainer, is a tool that makes training multi-turn agents possible and easy. If you’re already using Unsloth for GRPO and need to train agents that can handle complex, multi-turn interactions, ART simplifies the process.

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-c97e63a69ecd17685c6e09cb41fd625d43a3545d%2FScreenshot_2025-07-19_at_1.23.18_PM.webp?alt=media" alt="" width="375"><figcaption><p>Agent models trained with Unsloth+ART are often able to outperform prompted models on agentic workflows.</p></figcaption></figure></div>

### ART + Unsloth

ART builds on top of Unsloth’s memory- and compute-efficient GRPO implementation. In addition, it adds the following capabilities:

#### 1. Multi-Turn Agent Training

ART introduces the concept of a “trajectory”, which is built up as your agent executes. These trajectories can then be scored and used for GRPO. Trajectories can be complex, and even include non-linear histories, sub-agent calls, etc. They also support tool calls and responses.

#### 2. Flexible Integration into Existing Codebases

If you already have an agent working with a prompted model, ART tries to minimize the number of changes you need to make to wrap your existing agent loop and use it for training.

Architecturally, ART is split into a “frontend” client that lives in your codebase and communicates via API with a “backend” where the actual training happens (these can also be colocated on a single machine if you prefer using ART’s `LocalBackend`). This gives some key benefits:

* **Minimal setup required**: The ART frontend is has minimal dependencies and can be easily added to existing Python codebases.
* **Train from anywhere**: You can run the ART client on your laptop and let the ART server kick off an ephemeral GPU-enabled environment, or run on a local GPU
* **OpenAI-compatible API**: The ART backend serves your model undergoing training via an OpenAI-compatible API, which is compatible with most existing codebases.

#### 3. RULER: Zero-Shot Agent Rewards

ART also provides a built-in general-purpose reward function called [RULER](https://art.openpipe.ai/fundamentals/ruler) (Relative Universal LLM-Elicited Rewards), which can eliminate the need for hand-crafted reward functions. Surprisingly, agents RL-trained with the RULER automatic reward function often match or surpass the performance of agents trained using hand-written reward functions. This makes getting started with RL easier.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-b4168c3c380f6c8258c31083dcb65a5fcd2308b8%2FScreenshot_2025-07-19_at_1.21.08_PM.webp?alt=media" alt="" width="375"><figcaption></figcaption></figure>

```python
# Before: Hours of reward engineering
def complex_reward_function(trajectory):
    # 50+ lines of careful scoring logic...
    pass

# After: One line with RULER
judged_group = await ruler_score_group(group, "openai/o3")
```

### When to Choose ART

ART might be a good fit for projects that need:

1. **Multi-step agent capabilities**: When your use case involves agents that need to take multiple actions, use tools, or have extended conversations
2. **Rapid prototyping without reward engineering**: RULER’s automatic reward scoring can cut your project’s development time by 2-3x
3. **Integration with existing systems**: When you need to add RL capabilities to an existing agentic codebase with minimal changes

### Code Example: ART in Action

```python
import art
from art.rewards import ruler_score_group

# Initialize model with Unsloth-supported basemodel
model = art.TrainableModel(
    name="agent-001",
    project="my-agentic-task",
    base_model="Qwen/Qwen2.5-14B-Instruct",  # Any Unsloth-supported model
)

# Define your rollout function
async def rollout(model: art.Model, scenario: Scenario) -> art.Trajectory:
    openai_client = model.openai_client()
    trajectory = art.Trajectory(
        messages_and_choices=[
            {"role": "system", "content": "..."},
            {"role": "user", "content": "..."}
        ]
    )
    # Your agent logic here...    
    return trajectory

# Train with RULER for automatic rewards
groups = await art.gather_trajectory_groups(
    (
        art.TrajectoryGroup(rollout(model, scenario) for _ in range(8))
        for scenario in scenarios
    ),
    after_each=lambda group: ruler_score_group(
        group,
        "openai/o3",
        swallow_exceptions=True
    )
)

await model.train(groups)
```

### Getting Started

To add ART to your Unsloth-based project:

```bash
pip install openpipe-art # or `uv add openpipe-art`
```

Then check out the [example notebooks](https://art.openpipe.ai/getting-started/notebooks) to see ART in action with tasks like:

* Email retrieval agents that beat o3
* Game-playing agents (2048, Tic Tac Toe, Codenames)
* Complex reasoning tasks (Temporal Clue)


# Unsloth Updates

Unsloth Changelog for our latest releases, improvements and fixes.

To use the latest changes, [update Unsloth](/docs/new/studio/install#update-unsloth-studio).

{% updates format="full" %}
{% update date="2026-10-19" tags="new-releases" %}

## DeepSeek V4.1-Flash-1017 and FLUX.3

DeepSeek-V4.1-Flash-1017 and FLUX.3 Image Klein are here! Run both locally in Unsloth. Also new: images and videos in chat, multi-GPU training with FSDP2 (alpha) and one-command MCP setup for coding agents.

#### Highlights

* **DeepSeek-V4.1-Flash-1017**: 552B (16B active) - runs on \~125 GB at 1-bit with its Engram table on SSD [Guide](https://unsloth.ai/docs/models/deepseek-v4.1-flash)
* **FLUX.3 Image Klein**: Runs on 24 GB and edits with up to 10 reference images [GGUF](https://huggingface.co/unsloth/flux-3-image-klein-GGUF)
* **One-command MCP setup**: `unsloth mcp add` connects Claude Code, Codex or Cursor to Unsloth.
* **Images and videos in chat**: Local chat models use the image or video model loaded beside them.
* **Qwen-Image-Edit-2509 and 2511**: Edit with up to 3 images at once.
* **Token preview**: See what your model will learn before training starts.

<a href="https://github.com/unslothai/unsloth/releases/tag/v0.1.909-beta" class="button primary">Full Release notes</a><a href="/docs/models/deepseek-v4.1-flash" class="button secondary">DeepSeek-V4.1-Flash</a>
{% endupdate %}

{% update date="2026-10-15" tags="new-releases" %}

## NVIDIA Nemotron 3.5 + Meta Muse Spark Open

Run NVIDIA's new Nemotron 3.5 models and Meta's new Muse Spark Open locally in Unsloth. New: text-diffusion LLMs, DeepSpeed training and better prompt caching for coding agents.

#### Highlights

* **NVIDIA Nemotron 3.5 Bolt**: Run on 24 GB of memory. [Guide](https://unsloth.ai/docs/models/nemotron-3.5-bolt)
* **NVIDIA Nemotron 3.5 Super VL**: 120B (12B active) [Guide](https://unsloth.ai/docs/models/nemotron-3.5-super-vl)
* **Meta Muse Spark Open**: 816B (42.7B active) - runs on 256GB [Guide](https://unsloth.ai/docs/models/muse-spark-open)
* **Better prompt caching for coding agents**: Claude Code, Codex etc. on local GGUF models re-process far less of each prompt.
* **Browser use**: Always allow properly works, cursor, and separate tabs for each chat.
* **Images through your connections**: generate with sd-server, Ollama, Lemonade and other image servers.
* **Diffusion LLMs**: train and chat with LLaDA, Dream, SDAR and more, with a live denoising canvas.

<a href="https://github.com/unslothai/unsloth/releases/tag/v0.1.908-beta" class="button primary">Full Release notes</a><a href="/docs/models/nemotron-3.5-bolt" class="button secondary">Nemotron 3.5 Bolt</a><a href="/docs/models/muse-spark-open" class="button secondary">Meta Muse Spark Open</a>
{% endupdate %}

{% update date="2026-10-12" tags="new-releases" %}

## Browser Use + **Qwen-Image-2.1-Turbo**

Models can now use the browser in Unsloth. Turn on Browser use and your model can open pages, read them, click, type and scroll while you watch. Qwen-Image-2.1-Turbo is here too, making images in just 8 steps. Also included: text-diffusion LLM training, faster MoE GGUFs and speculative decoding on Mac.

#### Highlights

* **Browser use.** Local models can take control of the Unsloth Browser
* **Qwen-Image-2.1-Turbo**: images in 8 steps (2.5x faster). [See Guide](https://unsloth.ai/docs/models/qwen-image-2.1)
* **Train text-diffusion LLMs** like DiffusionGemma, LLaDA and Dream
* **Up to 3x faster MoE GGUFs** that are bigger than your VRAM
* **Speculative decoding on Mac** for up to 2.8x faster MLX replies

<a href="https://github.com/unslothai/unsloth/releases/tag/v0.1.906-beta" class="button primary">Full Release notes</a><a href="/docs/models/qwen-image-2.1" class="button secondary">Qwen Image 2.1</a>
{% endupdate %}

{% update date="2026-10-08" tags="new-releases" %}

## Train your own Decision model

Turn any text or vision LLM into a Jev-style decision model in Unsloth, with decision accuracy going from 30% to 80%. Train, test, export and serve decision models directly from Unsloth. Also included: native ComfyUI models, diffusion improvements and a better Browser in Desktop.

#### Highlights

* **Turn any model into a Jev-style decision model.** Accuracy 30% to 80%
* **Load ComfyUI diffusion models** natively in Unsloth
* **Faster + more accurate diffusion** with INT8 ConvRot and more
* **Better Browser in Desktop** with many bug fixes
* **Sandboxing** with Bwrap for Linux, Seatbelt for Mac and MXC for Windows

<a href="https://github.com/unslothai/unsloth/releases/tag/v0.1.904-beta" class="button primary">Full Release notes</a><a href="/docs/basics/train-your-own-decision-model-with-unsloth" class="button secondary">Decision Model Guide</a>

{% embed url="<https://github.com/user-attachments/assets/92959172-d2a2-450b-a5bf-6369a1e5ae16>" %}
{% endupdate %}

{% update date="2026-10-07" tags="new-releases" %}

## EmbeddingGemma 2 + Browser

This release adds a browser inside Unsloth (browser use coming very soon), so files, web pages and pages the model writes open right beside your chat. It also brings [EmbeddingGemma 2](https://unsloth.ai/docs/models/embeddinggemma-2), Google's new multimodal embedding model, plus new Audio pages and lower-memory training.

#### Highlights

* **Browser beside chat** for files, web pages and model-written HTML, as tabs
* **EmbeddingGemma 2**: run and fine-tune Google's new model for text, code, images, video and audio [Guide](https://unsloth.ai/docs/models/embeddinggemma-2)
* **Audio pages**: Speak, Clone, Music, Transcribe, Edit, Separate, Convert, with 70+ GGUF models
* **Keep multiple models loaded**, plus vLLM and SGLang engines (opt-in)
* **Sandbox** for code the model runs, now with a Settings tab and Colab and Docker support
* **Live preview** during image and video generation

<a href="https://github.com/unslothai/unsloth/releases/tag/v0.1.903-beta" class="button primary">Full Release notes</a><a href="/docs/models/embeddinggemma-2" class="button secondary">EmbeddingGemma 2</a>
{% endupdate %}

{% update date="2026-10-01" tags="new-releases" %}

## Command Palette + Desktop UI/UX

This release makes Unsloth Desktop faster and easier to use with improved navigation, shareable run settings, clearer errors, faster Laya decisions, expanded Decision API support, and better 4-bit LoRA training.

* **Cmd/Ctrl+P command palette** and shareable GGUF run settings.
* [**Laya decisions**](#laya-decision-models--library) **up to 4.1x faster** with hosted provider support.
* **Clearer errors** with improved load/generation messages and logs.
* **Lower-memory 4-bit LoRA training** for NVFP4, INT4, and MXFP4 checkpoints.

#### Desktop, chat, and models

* Share GGUF run settings without loading models.
* Continue responses and fix HTML canvas errors directly in chat.
* Improved model fit warnings, image loading, and desktop controls.
* Added `FastModel` fine-tuning support for T5, T5Gemma, BART, Marian, and more.
* Added `unsloth eval` for evaluating checkpoints and LoRA adapters.
* Improved multi-GPU training and generation support.

#### 4-bit checkpoint training

Pre-quantized checkpoints now train using their original packed weights, reducing memory use while preserving accuracy.

* Qwen3.8-27B-NVFP4 uses 40.2 GB peak memory instead of 72.9 GB.
* INT4 checkpoints stay packed instead of being re-quantized.
* Kimi-K2.7-Code and gpt-oss retain packed experts during LoRA training.

#### Decision API, image, and platform updates

* Laya decision models respond up to 4.1x faster, with hosted providers available through Connections.
* Image models preserve INT8/FP8 during offloading for faster generation.
* Qwen-Image-2.1 renders 1536×1536 images up to 3.6x faster on Radeon 8060S.
* Added AMD Windows support for RX 5000–9000 GPUs.
* Updated MLX support and Mac LoRA exports to PEFT or GGUF.

<a href="https://github.com/unslothai/unsloth/releases/tag/v0.1.902-beta" class="button primary">Full Release notes</a><a href="/docs/models/decision-laya" class="button secondary">Laya Decision Models Guide</a>
{% endupdate %}

{% update date="2026-09-28" tags="new-releases" %}

## Laya Decision Models + Library

We're adding support for decision models, a unified Library for docs and media, document viewer, many Apple Silicon improvements, creation of Skills, and \~4.5× faster image and video generation.

* Run and serve **Decision Models** like Laya (open-source Jev) locally through a TypeSafe-compatible API.
* **Skills Editor** to create, edit, and delete Skills directly in Desktop.
* **ModelScope** model downloading is now available for users who can't use Hugging Face
* **Document Viewer** for PDF, Word, Excel, PowerPoint, and chats.
* New **Library**: Manage chats, images, videos, and files from the new **Library** tab.
* **Apple Silicon Improvements** including batched serving, structured outputs, and TurboQuant KV cache.
* Configure how much of a model each GPU receives.
* **Faster Image + Video Generation** with \~4.5x faster LTX-2.3 clips and 1.7–6.3x faster VAE decoding.

<a href="https://github.com/unslothai/unsloth/releases/tag/v0.1.901-beta" class="button primary">Full Release notes</a><a href="/docs/models/decision-laya" class="button secondary">Laya Decision Models Guide</a>
{% endupdate %}

{% update date="2026-09-22" tags="new-releases" %}

## Qwen-Image-2.1 + Skills

You can now run Qwen-Image-2.1 locally with Unsloth! This release also includes custom Agent Skills, and easier chat/project management. It also brings 2x faster reasoning blocks (60 FPS vs 30 FPS), more reliable training, and improved Linux installs and updates.

<a href="/docs/models/qwen-image-2.1" class="button primary">Qwen-Image-1.2 Guide</a>

* **Qwen-Image-2.1 support:** Run Qwen-Image-2.1 locally with Unsloth for image generation.
* **Custom Agent Skills:** Add task-specific skills, reuse skills from Claude Code and `.agents` folders, and select them in chat with `@`.
* **Smoother chat experience:** Long reasoning blocks now render at 60 FPS, up from 30 FPS. Includes a redesigned thinking UI, smoother code streaming, and more reliable model loading and chat settings.
* **Easier chat and project management:** Drag chats to reorder, pin, or move them into projects across all platforms. Edit project names, instructions, and folders directly from the Projects page.
* **Better model and storage handling:** Vision models can see images returned by MCP tools. Reuse downloaded models without fetching duplicate copies, and inspect or clear caches from Settings.
* **More reliable training and exports:** Resume past training runs with accurate remaining-time estimates. Fixes improve Hugging Face dataset recipes, column mapping, and split/subset selection. Hub uploads exclude files left over from earlier exports.
* **Improved Linux and AMD support:** In-app updates for Debian installs, a native ARM64 installer for Ubuntu 24.04+, and a new AMD ROCm Docker image with Unsloth Studio and JupyterLab.
  {% endupdate %}

{% update date="2026-09-17" tags="new-releases" %}

## Docker + MultiUser + AMD Support

We're releasing our new updated Docker image along with multi-user accounts, RDNA1+2, FP8/INT8 diffusion support, ARM64 CUDA Windows support and many training, GRPO and inference improvements

#### Highlights:

* Updated Docker with NVIDIA and AMD support: [Guide](https://unsloth.ai/docs/get-started/install/docker)
* Multi-user accounts with isolation: **Settings > Accounts**
* INT8/FP8 image diffusion inference: 2x faster
* ARM64 Windows CUDA support for training and inference
* GRPO: Qwen3.5 and latest TRL/vLLM support
* AMD RDNA1 and RDNA2 support
* Better Windows NVIDIA GPU detection and recovery

#### Updated Docker image

* Updated CUDA image and Studio setup, removed bundled caches, and restored Unsloth’s training patches on GPU hosts.
* Studio data persists on a volume without pinning app code. Improved updates, generated passwords, configurable ports, and package preservation.
* Added an AMD ROCm image for supported Linux hosts.

#### Multi-user accounts

* Create accounts in **Settings > Accounts** with one-time setup codes and individual passwords.
* Accounts keep work separate while sharing a loaded model when settings match. Single-account behavior is unchanged.

#### Chat + reasoning

* Edit, reorder, and steer queued prompts, even while a local model loads.
* Added GGUF reasoning budgets, adjustable chat width, desktop scaling, and a compact composer.
* Improved long-reasoning responsiveness, chat-history preservation, and tool-generated file and image handling.

#### Hardware + inference

* MLX gains video input, optional MoE and decode optimizations, and more reliable multimodal chats.
* Improved GGUF memory estimates, DGX Spark handling, NVIDIA detection, NVLink checks, and user-selected GPU ordering.
* Added native Windows ARM64 desktop packaging, Qwen3.5 `fast_inference`, and Ascend NPU detection.

#### Research + API

* Deep Research better preserves unfinished reports, handles JSON output, and flags runs without evidence. Scanned PDF uploads gain a local OCR fallback.
* Improved provider reasoning controls, MCP compatibility, tool calling, and streamed error reporting.
* Added `/v1/chat/completions` video input and API access to installed Ollama models.

#### Training + installation

* Added Data Recipes dataset downloads and fixed 16-bit fine-tune exports, LoRA saves, and Push to Hub.
* Improved dataset handling, training memory estimates, and checkpoint saves on container shutdown. Core fixes cover normalization, RoPE, Q-GaLore, and distributed device selection.
* Improved Windows installs, GPU package repair, dependency caching, and verified offline installs.

#### Security + tool execution

* Strengthened credential protection and tool-call validation, with approval requests for file access outside the sandbox.
* Added preview OS-level tool sandboxing on supported Linux and macOS hosts. Network access remains unrestricted; automatic mode keeps existing safeguards when OS isolation is unavailable.

<a href="/docs/get-started/install/docker" class="button primary">Docker Guide</a>
{% endupdate %}

{% update date="2026-09-02" tags="new-releases" %}

## 2x Faster Qwen3.8-Flash + GLM-5.3

Run [**Qwen3.8-Flash-Next**](/docs/models/qwen3.8-next) and [**GLM-5.3-Flash**](/docs/models/glm-5.3-flash) up to **2× faster** with faster decoding and bonus MTP, now enabled by default. This release also includes **170+ improvements** across training, chat, hardware support, APIs, and performance.

* Faster Qwen and GLM generation with MTP, plus improved tool use and recommended Qwen settings.
* Much less laggy UI/UX for all chats especially on long conversations
* New support for **MiniMax-Music3, Higgs, and MOSS** audio models, including progress tracking and clip management.
* More reliable model loading across local servers and connected providers.
* Safer chat editing that preserves tool calls, replies, media, and conversation branches.
* Improved multi-GPU training, memory fitting, model placement, and GGUF exports.
* Stronger AMD/ROCm installation, detection, and GPU compatibility.
* More reliable MCP, Deep Research, OAuth, parallel tool calls, and agent workflows.
* New OpenAI-compatible APIs for **video, audio, and MLX-served models**.
* Improved desktop updates, LAN port controls, downloads, and GPU memory cleanup.

<a href="/docs/models/qwen3.8-next" class="button primary">Qwen3.8-Flash-Next</a><a href="/docs/models/glm-5.3-flash" class="button primary">GLM-5.3-Flash</a><a href="/docs/models/glm-5.3" class="button primary">GLM-5.3</a>
{% endupdate %}

{% update date="2026-08-27" tags="new-releases" %}

## Qwen3.8-Flash-Next + GLM-5.3

[Qwen3.8-Flash-Next](/docs/models/qwen3.8-next), [GLM-5.3-Flash](/docs/models/glm-5.3-flash) + [GLM-5.3](/docs/models/glm-5.3) can now run locally in Unsloth!

* Qwen3.8-Flash runs on 75GB RAM
* GLM-5.3-Flash runs on 102GB RAM + VRAM
* Up to 5x faster RAM offloading
* Repeated Auto Compaction now works reliably
* 100+ chat, reliability and performance improvements

<a href="/docs/models/qwen3.8-next" class="button primary">Qwen3.8-Flash-Next</a><a href="/docs/models/glm-5.3-flash" class="button primary">GLM-5.3-Flash</a><a href="/docs/models/glm-5.3" class="button primary">GLM-5.3</a>

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1YneA51oWzaIiwep9H5I%2F1000024423.gif?alt=media&amp;token=40a169cc-12f1-403a-898c-b310f81ff52a" alt="" width="375"><figcaption></figcaption></figure></div>

#### Qwen3.8-Flash-Next

125B multimodal reasoning model and early preview of Qwen4's architecture.

* 1-bit Dynamic GGUF runs on 75GB RAM or unified memory
* Up to 262K context with text and images
* None, Low, Medium and Extra High reasoning modes
* Preserved Thinking improves consistency in long chats

#### GLM-5.3-Flash

320B multimodal model with 18B active parameters.

* 1-bit model runs on 102GB combined RAM + VRAM
* Up to 1M context for text, images and documents
* Low, High and Max reasoning modes
* Improved coding, agent and vision performance

#### Unsloth improvements

* Large GGUFs automatically split across GPU and system RAM
* See memory estimates before loading
* Chats recover after disconnects
* Better multi-image vision and MCP image support
* Export chats as JSONL
* Improved Auto Compaction controls
* Fixes for Linux, NVIDIA Wayland, AMD and Windows

To run or train the new models, you can download Unsloth Desktop:

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth Desktop</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)
  {% endupdate %}

{% update date="2026-08-20" tags="new-releases" %}

## Auto compaction + LAN Access

Thanks for the support for Qwen3.8-27B and Unsloth Desktop last week! For our [new `v0.1.801-beta` release](https://github.com/unslothai/unsloth/releases/tag/v0.1.801-beta), we merged 200+ PRs to introduce many new features, fixes including:

* **Auto Compaction (Experimental)** for longer chats beyond context limits
* [**Remote & LAN Access (Preview)**](/docs/basics/lan) for easy network access without Cloudflare links
* **Faster Chat** - Improved streaming performance, reduced UI lag, and smoother long conversations.
* Support for **custom llama.cpp builds**. Toggles for Cache RAM, Mmap, Mlock, Checkpoints, Spe Decoding KV Cache, Vision On / Off
* [Unsloth Dynamic v3.0](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs) is released. New Qwen3.8-27B Dynamic v3.0 GGUFs deliver >10% higher top-1 accuracy compared to everyone else. Works with Unsloth.

#### Auto compaction (Experimental)

Long chats can now exceed context limits by moving older turns into a searchable archive.

* Older turns are removed only when needed and remain searchable.
* Fresh context epochs improve recall without permanently trimming chats.
* Uses retrieval instead of summarization for better accuracy.

#### Remote & LAN access (Preview)

Access Unsloth from other devices on your network.

* New remote access settings.
* LAN control, QR codes, and auto-start options.
* Disabled by default for security.

#### Chat improvements

* Faster long chats and improved threading.
* Projects organize chats, files, and workspaces.
* Added prompt queueing, shortcuts, `edit_file`, and better tool support.

#### Hardware, inference, API

* Support for custom llama.cpp builds and Intel XPU.
* More inference controls (cache, mmap, mlock, checkpoints, speculative decoding, vision).
* Improved GPU validation, VRAM handling, and compatibility.
* Structured outputs in Responses API.
* Better llama-server recovery.

To run or train Qwen3.8, you can download Unsloth Desktop:

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth Desktop</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)
  {% endupdate %}

{% update date="2026-08-14" tags="new-releases" %}

## Qwen3.8

Qwen3.8-27B and Qwen3.8-2.4T can now be run locally in Unsloth!

Run on 17GB RAM via Unsloth Dynamic GGUFs. You can also fine-tune Qwen3.8-27B in Unsloth. Qwen3.8-27B is by far the strongest model for its size. We also uploaded NVFP4 quants.

<a href="/docs/models/qwen3.8" class="button primary">Run Qwen3.8 Guide</a><a href="/docs/models/muse-glimmer/train" class="button secondary">Fine-tune Muse Glimmer</a>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FSqxs6NjShWrLfRKhDy1m%2Fvolcano%202.gif?alt=media&amp;token=395274a0-b437-403a-8a01-8e8502f9d225" alt="" width="375"><figcaption></figcaption></figure>

Other Unsloth updates include:

* Qwen3.8-27B + extra llama-server arguments allowed + custom VRAM toggle
* External provider has tool calling + tool support + login with Codex
* Fast FP8 10x faster MiniMax-H3 inference (3 minutes vs 30)
* 10% faster inference for GGUFs + Bypass permissions fixed
* Connected [API providers](/docs/integrations/connections) can use their own Search or Unsloth Desktop's built-in Search and tools. Tool results are passed back to the model so it can continue multi-step tasks.
* Sign in with a Codex subscription and use Codex tools inside Chat.

To run or train Qwen3.8, you can download Unsloth Desktop:

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth Desktop</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)
  {% endupdate %}

{% update date="2026-08-11" tags="new-releases" %}

## Introducing Unsloth Desktop

Introducing Unsloth Desktop 🦥 - the first desktop app to run and train models locally.\
Open-source. Runs on Mac, Windows and Linux

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FyUdniXManojk20Q3M5Lf%2Fintroducing%20unsloth%20desktop%20thumb.png?alt=media&amp;token=5ecd2f25-b996-4749-8242-ffd21b080ae7" alt=""><figcaption></figcaption></figure>

• Supports MLX, diffusion image/video, audio, GGUF\
• Connect Claude Code and Codex to local LLMs\
• 50% more accurate, self-healing tool calls + sandboxed code exec\
• Works for CPU + multiGPU setups - NVIDIA, AMD, Intel, Mac\
• Train models 2× faster with 70% less VRAM\
• Private web search, deep research, RAG, MCP + exports (NVFP4, GGUF)\
• Use Unsloth’s OpenAI-compatible API and cloud models\
• Securely deploy LLMs remotely and access anywhere

You can download Unsloth Desktop now:

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth Desktop</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)
  {% endupdate %}

{% update date="2026-08-10" tags="new-releases" %}

## Meta Muse Glimmer is here!

Meta has released Muse Glimmer, the first open model from Meta Superintelligence Labs. Muse Glimmer is a dense, 30B parameter model by Meta, built for local agentic and coding workflows. Released under the Apache 2.0 license you can run Muse Glimmer 30B via Unsloth and Unsloth Dynamic quants for the best performance.

<a href="/docs/models/muse-glimmer" class="button primary">Run Muse Glimmer</a><a href="/docs/models/muse-glimmer/train" class="button secondary">Fine-tune Muse Glimmer</a>

Muse Glimmer 30B can run locally on **20GB RAM/VRAM** setups, including Mac and GPUs. You can also train and fine-tune Muse Glimmer 30B using Unsloth on **20GB VRAM**.

You can run and fine-tune Muse Glimmer via Unsloth Desktop app:

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth Desktop</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)
  {% endupdate %}

{% update date="2026-07-29" tags="new-releases,v0.1.51-beta" %}

## Kimi K3 + Deep Research + Parallel Chat

Hey everyone! [Kimi K3](/docs/models/kimi-k3) can now run locally with Unsloth Dynamic GGUFs, Unsloth can keep multiple chats generating in parallel, and the new Deep Research mode plans, reads and cites sources using your local model.

This release also brings better [AMD](/docs/basics/amd) and Intel GPU support, DoRA training, and many installer, MLX, export and inference fixes.

#### Kimi K3

Moonshot AI’s **Kimi K3** is a 2.8T-parameter MoE model with 104B active parameters, native vision support and a 1M context window. Kimi K3 is thinking-only and Unsloth supports low, high and max reasoning efforts.

You can run our [Kimi K3 Dynamic GGUFs](https://huggingface.co/unsloth/Kimi-K3-GGUF) through Unsloth. Unsloth automatically detects multi-GPU setups and can offload model layers to system memory.

Kimi K3 is a very large model, so plan your hardware accordingly:

* `UD-IQ1_S` is 595GB in disk space.
* `UD-Q4_K_XL` is 1.51TB in disk space.
* For lossless inference, use `UD-Q8_K_XL`, which is 1.56TB in disk space.

Read our full [Kimi K3 guide](https://unsloth.ai/docs/models/kimi-k3) and learn more about [Unsloth Dynamic 2.0 GGUFs](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs).

#### Parallel Chat

Unsloth can now run multiple conversations at once. Starting a **New Chat** leaves the previous answer generating, and each active conversation gets its own progress indicator and Stop control.

* 4 llama-server slots by default, adjustable in the web UI.
* Tools, uploads, self-healing and agents stay isolated between chats.
* Stop one chat without interrupting others or restarting the server.
* Unsloth reduces the slot count when memory is limited.
* Reloading the model still stops active chats after confirmation.

#### Deep Research

Deep Research turns a local model into a complete research workflow. Give it a question and it will create a plan, search the web, organize the evidence and produce a cited report.

* Review and edit the plan before research begins.
* Follow its progress and collected sources while it works.
* Resume or cancel without losing completed research.
* Allow or block websites to control source selection.
* Sources and citations stay saved with each run.
* Before writing, Unsloth checks for unsupported claims, contradictions and unresolved gaps. Recommendations without direct evidence are marked as testable inferences.

One run can be active per chat. Deep Research currently works with local models. Optional full-page grounding can read top search results into temporary RAG before synthesis; enable it with `UNSLOTH_RESEARCH_AUTO_SCRAPE=1`. It remains off by default because it adds scraping time and needs additional context.

#### Improved AMD Support

This update builds on our initial [AMD release and setup guide](https://unsloth.ai/docs/basics/amd) with support for more GPUs and more reliable ROCm inference and training.

* Improved detection for RDNA2, Radeon, Ryzen, Strix Halo and workstation GPUs.
* MI50 and Radeon VII support 16-bit LoRA and full fine-tuning on Linux.
* Fixed 4-bit NaNs, library conflicts and long RDNA startup stalls.
* Unsupported Windows HIP GPUs can fall back to Vulkan.
* Vulkan devices show their real names and can be selected individually.
* Clean Windows installs no longer require Winget or developer tools.

#### Training, Models and Platform Updates

* Intel XPU enables local chat and training on Intel Arc and Data Center GPUs.
* DoRA is available alongside LoRA and full fine-tuning.
* Large exports can use all visible GPUs to avoid GPU 0 memory limits.
* MLX adds streaming datasets, continued pretraining, callbacks and better VLM and LoRA support.
* Fixed incorrect `flash_attention_2` model output.
* Unsloth and saving utilities now work without bitsandbytes.
* Fixed `.json` datasets and Qwen3.5/3.6 MoE and GRPO notebook setup.
* Hub download folders are easier to find and open.
* Release notes are available in the Unsloth update popup.
* `unsloth start --as-subagent` lets Codex, Claude Code and Pi delegate tasks to a local Unsloth model.
  {% endupdate %}

{% update date="2026-07-20" tags="new-releases,v0.1.50-beta" %}

## AMD support is here!

Hey everyone! This release brings local LLM training and inference to [AMD GPUs](/docs/basics/amd) across Windows, WSL and Linux.

<a href="/docs/basics/amd#installing-unsloth-on-amd" class="button primary" data-icon="bolt">Quickstart</a><a href="/docs/basics/amd#id-2x-faster-training-and-50-more-accurate-tool-calls" class="button secondary" data-icon="star">Features</a><a href="https://github.com/unslothai/unsloth" class="button secondary" data-icon="github">Github</a>

Starting today, our AMD collaboration, custom Triton kernels, and math algorithms enables you to train and run 500+ models across AMD's Radeon, Instinct, Vulkan, Ryzen and data center GPUs, up to 2× faster with 70% less VRAM and no accuracy loss. Optimized ROCm builds also support GGUF & Safetensors inference.

<img src="https://github.com/user-attachments/assets/bb8bc525-9fb8-405d-98f5-6c9c07128085" alt="" width="375">

#### July 23 Update

1. Added **RDNA2, Gorgon Halo, Vulkan support** + fixed AMD installing not detecting GPUs on Strix Halo / other AMD GPUs
2. Better RDNA4, HIP / ROCm failure auto fixing and catching
3. **2x faster unified memory** AMD safetensors loading + much faster gradient checkpointing for unified memory devices
4. Added **voice dictation / whisper.cpp** preliminary support for fast text to speech
5. Fixed rollback environments during installs eating 5GB of disk space - now auto cleans

#### Train LLMs Locally on AMD

* Train, run RL, chat with and deploy models locally on AMD GPUs.
* More reliable AMD GPU detection and installation across Windows, WSL and Linux.
* Improved ROCm compatibility for AMD MI300X and MI325X GPUs.
* Remote access Unsloth via `unsloth studio --secure` through free HTTPS via Cloudflare

#### Run Larger Models on Your Hardware

* Use automatic GPU placement or choose exactly which GPUs and model layers to use.
* Move MoE expert layers into system memory to help larger models fit.
* Split models across multiple GPUs or use Tensor Parallelism.
* Save hardware settings separately for each model and quant.

#### Faster Chat Restarts and More Reliable Downloads

* Resume long chats without rebuilding the full conversation after an idle model frees its VRAM.
* Stalled Hugging Face XET downloads automatically retry over standard HTTP.
* Existing GGUF files are reused instead of being downloaded again whenever a model loads.

#### Better Search, Tools and Agents

* Web search can now read PDF papers, manuals and other PDF results.
* Parallel tool calls, reasoning output and tool retries work more reliably.
* A new opt-in MCP endpoint lets compatible AI clients inspect models and training history, start or stop training, load checkpoints, validate recipes and export GGUFs.
  * Enable it with `UNSLOTH_STUDIO_ENABLE_MCP=1` and set the required bearer token with `UNSLOTH_STUDIO_MCP_TOKEN`.

#### Training and Export Fixes

* Text-only training with multimodal models no longer truncates long examples before packing.
* Fine-tuned Qwen3.5 and Qwen3.6 MTP models now export correctly to GGUF.
* Fixed a Windows permission error that could stop GGUF exports during the final write step.

#### In Case You Missed It

Our previous Studio release added Dynamic NVFP4 models, deeper personalization, seven new display languages, safer agents and Vulkan GPU acceleration.

**Dynamic NVFP4:**

Unsloth Dynamic NVFP4 keeps accuracy-sensitive layers in FP8 or BF16 while running the rest in W4A4. On NVIDIA Blackwell GPUs, this enables up to 2.5x faster inference, while calibrated FP8 KV caches provide up to 2x longer context.

Read the [Dynamic NVFP4 guide](https://unsloth.ai/docs/basics/nvfp4) and explore our expanded [NVFP4 collection](https://huggingface.co/collections/unsloth/nvfp4), including Qwen3.6, Qwen3.5, Inkling, GLM-4.7 Flash and Gemma 4.

**Personalize Your Studio:**

Choose between Standard, Classic and Minimal color palettes, each with light and dark modes. You can also customize colors, import fonts and adjust font size, contrast, reduced motion and more.

Unsloth also includes a Voice settings tab for dictation, custom dictionary settings and read-aloud.

**New Languages:**

Unsloth Studio is now available in French, German, Spanish, Hindi, Arabic, Russian and Korean, alongside Chinese (Simplified), Japanese and Portuguese (Brazil). Browser-language auto-detection is now the default.

**Safer Agents:**

A four-level tool-call permission selector-**Ask**, **Approve for me**, **Off** and **Full access**-provides finer control over agents. Agent workspace isolation and safer installer checks also reduce the risk of unintended changes.
{% endupdate %}

{% update date="2026-07-15" tags="new-releases,v0.1.49-beta" %}

## Personalization, NVFP4, languages

Hey guys we got lots of new update for Unsloth, especially customization. Unsloth now yours to personalize: three color palettes plus custom colors and fonts, seven new display languages, and a new Voice settings tab for dictation and read-aloud. Agents get safer with a four-level tool-call permission selector (Ask, Approve for me, Off, Full access) and workspace isolation, and Intel GPUs finally get GPU-accelerated inference through new Vulkan `llama.cpp` support.

We’ve also added native support for [Inkling](https://unsloth.ai/docs/models/inkling), a 975B-parameter open model with 41B active parameters and up to a 1M-token context window. Licensed under Apache 2.0, it accepts text, images, and audio and generates text.

#### Dynamic NVFP4

We expanded our [NVFP4 collection](https://huggingface.co/collections/unsloth/nvfp4) with quantized versions of Qwen3.6, Qwen3.5, Inkling, GLM-4.7 Flash, and Gemma 4.

Read the [Dynamic NVFP4 guide](https://unsloth.ai/docs/basics/nvfp4) for details.

Personalize Your Unsloth

Unsloth is no longer limited to light and dark mode:

* Choose from **Standard**, **Classic**, and **Minimal** palettes, each with light and dark variants.
* Customize accent, background, and foreground colors.
* Import your own UI, heading, chat, and code fonts.
* Adjust font size, contrast, motion, cursor behavior, and font smoothing.
* Search settings and sync preferences across devices.

Light and dark modes keep their own customization values.

#### Seven New Languages

Unsloth now supports French, German, Spanish, Hindi, Arabic, Russian, and Korean, alongside Chinese, Japanese, and Portuguese. Browser-language auto-detection is now the default.

#### Voice Settings

A new Voice tab adds controls for dictation, pronunciation dictionaries, and read-aloud. Speech-to-text and text-to-speech support are coming soon; full voice conversations are still in development.

#### Safer Agents and Tool Calls

The old bypass toggle is now a four-level permission selector:

* **Ask:** approve every tool call.
* **Approve for me:** automatically run read-only actions and pause for potentially unsafe ones.
* **Off:** disable tool calls.
* **Full access:** allow all calls and disable the sandbox.

These controls cover terminal, Python, web search, RAG, and MCP tools. Agents also gain isolated workspaces, resumable sessions with `--persist`, safer remote-installer warnings, live tool-output streaming, and improved web extraction.

#### Vulkan and `llama.cpp`

The new Vulkan `llama.cpp` backend gives Intel GPUs GPU-accelerated inference instead of falling back to the CPU. It supports existing VRAM management, automatic context sizing, multi-GPU selection, and layer offloading.

AMD users can opt in with:

```bash
UNSLOTH_FORCE_VULKAN=1
```

We also improved update reliability, model-state recovery, and interruption of stalled generations.

#### Models and Training

This release also includes:

* Native Inkling support and multi-GPU B200 improvements.
* DeepSeek-V4 eager attention and trainable FP8 grouped experts.
* Reliable completion-only training through automatic chat-template marker detection.
* Correct handling of disabled gradient checkpointing.
* Improved RoPE scaling and context extension.
* Force-stopping for stuck training runs.
* Automatic routing for new model architectures and newer Transformers releases.
  {% endupdate %}

{% update date="2026-07-07" tags="new-releases,v0.1.48-beta" %}

## DeepSeek-V4 + NVFP4

Unsloth can now export NVFP4, FP8, and imatrix GGUFs after training; act as a llama-swap API system; add Japanese and Brazilian Portuguese support; and includes MLX, safetensors, tool calling, healing support, and more. Unsloth core makes GRPO 1.3x faster, adds HTTP fallback for stalled downloads, improves offline mode, speeds up MoE training by 3-5x, and fixes many bugs. This release series uses `unsloth>=2026.7.1`.

[DeepSeek-V4-Flash](/docs/models/deepseek-v4) is now supported with Thinking toggles and our improved chat template fixes.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FChQx2DGoDLyhf7mp8G2d%2Fdeepseek%20v4%20flashhh.png?alt=media&amp;token=a84f2fbc-8ab0-44cf-a3a8-0bdaaf80a2a7" alt="" width="375"><figcaption></figcaption></figure></div>

#### Smarter OpenAI-Compatible API Serving

Run one local API endpoint with safer model swapping and better agent-tool recovery.

* API requests can opt into automatic switching between downloaded local GGUFs, while unknown model names safely keep using the current model.
* `/v1/models` now returns clean model IDs and the local GGUF catalog instead of local `.gguf` paths.
* Idle auto-unload can free VRAM after inactivity, and tool-call healing can now be controlled per request.

#### Export Improvements

Exports are more flexible and avoid unnecessary downloads.

* Select multiple export formats at once, including portable FP8/INT8, GGUF LoRA, source-matched exports, imatrix GGUF, and compressed FP8/FP4.
* Multi-checkpoint exports avoid more repeated base-model downloads.
* FP8, INT8, and GGUF-LoRA exports now respect `trust_remote_code`, and GGUF export handles missing quantization settings more reliably.

#### RAG and File Chat

File chat is more useful on real documents.

* RAG attachments can now use whole-document context, with customizable embedding models and Hugging Face search.
* File chat reads more PDFs and Word documents correctly, including right-to-left text, Indic text, and DOCX tables.
* Local RAG checks are more reliable behind proxy setups.

#### Unsloth Polish and Reliability

Everyday use should feel smoother and more stable.

* Long training and chat runs are less likely to freeze silently.
* Compare mode, model switching, model cancellation, Hub browsing, and Hub Discover are more reliable.
* Project exports, chat exports, settings, guided tours, file dialogs, update screens, and reasoning UI are cleaner and more consistent.

#### Installer, Hardware, and Platform Fixes

Unsloth installs and runs more reliably across platforms.

* macOS installs no longer require CMake or Homebrew when a prebuilt `llama.cpp` is available, and Apple Silicon support handles paths with spaces and unified memory sizing better.
* Windows startup, UTF-8 handling, ROCm RAG embedding, and ROCm-on-WSL GPU support were improved.
* Blackwell GPU prebuilt selection, GGUF fit checks, tensor parallelism for vision/mmproj GGUFs, and local `llama.cpp` reuse are now more reliable.

#### Training, Models, and Kernels

Training and model loading are more reliable across more setups.

* GRPO now supports sequence packing by default, avoids repeated shared prompts, and handles DDP logit scaling correctly.
* Full fine-tuning, RL precision settings, gradient checkpointing, DDP RoPE buffers, and MoE LoRA detection were fixed.
* FP8 quantization/dequantization, Llama 3 RoPE scaling with Transformers v5, PEFT 0.19 LoRA reloads, and `fast_generate` error messages were improved.
  {% endupdate %}

{% update date="2026-06-18" tags="new-releases,v0.1.47-beta" %}

## GLM 5.2 + Hub + 3x longer contexts

[GLM-5.2](/docs/models/glm-5.2) is now supported in Unsloth Studio! All reasoning levels supported. **3x longer context lengths** are now achievable with our new auto fit algorithm with MTP, allowing longer chats. Bypass permissions mode, forkable chats, queue-able chats, a new hub for model discovery, parallel modules + HTTPS Cloudflare support and more! Use `unsloth studio --secure` for secure HTTPS global access!

<div data-with-frame="true"><img src="https://github.com/user-attachments/assets/93c18616-415f-48ea-957d-9e0fa97a45dd" alt="" width="563"></div>

#### Better context length algorithm

As per [PR 1](https://github.com/unslothai/unsloth/pull/6312) and [PR 2](https://github.com/unslothai/unsloth/pull/6447), we made Unsloth Studio's determination of memory usage and context length much better, achieving 3x longer context overall:

| Scenario                        | KV    | before  | after   |
| ------------------------------- | ----- | ------- | ------- |
| 1x 32GB pipeline (\~31 GB free) | f16   | 23,040  | 64,000  |
|                                 | q8\_0 | 43,520  | 114,944 |
|                                 | q4\_0 | 82,432  | 199,680 |
| 2x 32GB pipeline                | any   | 262,144 | 262,144 |
| 2x 24GB tensor (\~23 GB free)   | f16   | 134,049 | 262,144 |
|                                 | q8\_0 | 252,329 | 262,144 |

#### Chat Canvas, Forking & Queueing

* Edit assistant messages in place and re-run from any point in the thread.
* Fork a thread to branch a conversation without losing the original.
* Temporary (incognito) chats that leave nothing behind.
* Queue new prompts while a generation is still running instead of waiting.
* Chat "artifacts" are now **canvas**, with inline **HTML canvas cards** that auto-render, a Code view, and DiffusionGemma keeps its raw code visible inline instead of collapsing.
* Chat search now covers every message and surfaces your own messages first.

#### Hub (Redesigned)

* Full-page Hub with a trending feed, search, and custom model paths support.
* README preview in a split-view feed so you can read before you download.
* Downloads default to the faster **Xet** transport, with automatic HTTP fallback if a transfer stalls.
* New "Load on selection" toggle to set load options before a model loads.
* Google logo shown for DiffusionGemma and future Gemma derivatives.

#### Models & Inference

* DeepSeek-OCR and more vision models now load and run without errors.
* Fixed fast inference on the latest vLLM (0.22+) so speed-ups work again.
* Tensor parallelism is more reliable: if the faster MTP path fails, it now recovers on its own instead of crashing.
* DiffusionGemma now shows the image forming live as it denoises, with accurate speed stats.

#### Security & Cloudflare Encrypted Studios

* New `--secure` Cloudflare-only mode for end-to-end encrypted studios, with server-side tools staying enabled under `--secure`. Use `unsloth studio --secure`!
* Bypass Permissions mode to skip confirmations and disable the tool sandbox when you want it.
* Auto detect Hugging Face Virus scanning + dangerous files in repos.

#### Logging and API

* New **API server monitor** in Unsloth.
* Faster API calling and less latency
* Much better streamlined logs - now with throughput and latency and removed a lot of bloated logs.

#### Hardware & Backend

* Better support for Blackwell RTX 50X and 60X GPUs
* Fix silent downgrading to CPU and not GPU
* torchao version is now selected from the installed torch.
* Installer now auto-repairs a broken or CPU-only PyTorch install and warns on silent CPU fallback, across NVIDIA + AMD on Win/Linux/Mac/WSL.
* Frees the chat model's VRAM when training starts, but only when the GPU is actually tight (no needless reloads otherwise).
* If llama-server hard-crashes at startup, Unsloth now steps through a recovery ladder instead of just failing.

#### Training & General Fixes & Parallel Modules

* MLX training updates.
* Improved GRPO training reliability with vLLM.
* Training startup made more reliable, with clearer errors for invalid VLM batches.
* Unsloth now cleans up leftover backend processes more reliably after crashes, restarts, or interrupted shutdowns.
* Export, Chat, Training, Recipes are all individualized / compartmentalized! This means you can do all 4 in parallel now! You can chat / do inference while you wait for a training run or an export!

To update Unsloth or install a new Unsloth Studio, you must use:

**macOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endupdate %}

{% update date="2026-06-12" tags="new-releases,v0.1464-beta" %}

## DiffusionGemma + Gemma 4 MTP

Ensure you install the latest [`v0.1.464-beta`](https://github.com/unslothai/unsloth/tree/v0.1.462-beta) or `2026.6.7`. [DiffusionGemma](https://unsloth.ai/docs/models/diffusiongemma), [Gemma 4 MTP](https://unsloth.ai/docs/models/mtp) and [**MiniMax-M3**](https://unsloth.ai/docs/models/minimax-m3) are all now supported.

* Run and train [DiffusionGemma](https://unsloth.ai/docs/models/diffusiongemma) via [Unsloth Studio](https://unsloth.ai/docs/new/studio).
* [Gemma 4 MTP](https://unsloth.ai/docs/models/mtp) is here! Run [Gemma 4](https://unsloth.ai/docs/models/gemma-4) \~2x faster with MTP.
* Audio chat is now supported for Gemma 4 (`wav`, `mp3`, `m4a`, `flac`, `webm`).
* Preserve Think added to Gemma 4.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fszoidwwj70dLm5vBjpmL%2Fdiffusiongemma.gif?alt=media&amp;token=8ce359fd-f92c-4cf8-a491-06b04c505cb7" alt="" width="375"><figcaption></figcaption></figure>

#### Hub + Download Manager (Experimental)

* Added a new **Hub** page for browsing, downloading, and managing Hugging Face models and datasets.
* Unsloth can now detect models and datasets already on your machine and show them alongside downloaded assets.
* Downloaded [GGUF models](https://unsloth.ai/docs/basics/inference-and-deployment/saving-to-gguf) now have direct **Run / New Chat** actions.

#### RAG / Chat with Files (Experimental)

* Added [**Chat with Files**](https://unsloth.ai/docs/new/studio/chat) in Unsloth, letting you ask questions over your own documents and knowledge bases.
* Supports hybrid search, citations, PDF previews, per-thread documents, and a built-in `search_knowledge_base` tool.

#### New Update Button + Hardware Support

* Unsloth now uses constant fresh up to date [llama.cpp prebuilts](https://unsloth.ai/docs/new/changelog) across CUDA, ROCm, Windows, Linux, and macOS.
* Added an in-app **Update llama.cpp** button so users can update the local backend without reinstalling Unsloth.
* Improved Windows / WSL AMD support, [Strix Halo ROCm support](https://unsloth.ai/docs/get-started/install/amd), [Blackwell CUDA selection](https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth), and clearer installer messages.

#### Local Chat, Tools & API Compatibility

* Local [tool calling](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms) is more reliable, with better ordering of tool cards, fewer duplicate tool loops, and support for tool use with GGUF vision models.
* Improved [OpenAI-compatible API](https://unsloth.ai/docs/basics/inference-and-deployment/llama-server-and-openai-endpoint) and Anthropic-compatible API behavior for local Unsloth servers, including better errors, token usage, stop reasons, and [Claude Code compatibility](https://unsloth.ai/docs/basics/claude-code).

#### Training & Fixes

* Improved [MLX support](https://unsloth.ai/docs/new/studio/install) with better model labels, generation speed stats, and fixes for [VLM training](https://unsloth.ai/docs/basics/vision-fine-tuning).
* Fixed several [training](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide) and [dataset](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/datasets-guide) edge cases, including non-writable Hugging Face caches and custom dataset mappings.
* Added many UI polish fixes across chat, menus, model picker, dark mode, import/export, and settings.

To update Unsloth or install a new Unsloth Studio, you must use:

**macOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endupdate %}

{% update date="2026-06-03" tags="new-releases,v0.1.44-beta" %}

## Gemma 4 12B, New UI, MCP, Projects

This update focuses mainly on Gemma 4 12B, MCP, Projects, Canvas, CUDA 13.3 and the new chat UI. Next week we'll have an even bigger update.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoRn2YEPb9lJ81kP3GQQh%2Fwhats%20on%20ur%20mind.png?alt=media&amp;token=f388e42a-bb05-4a09-9cbf-ec699a6baf6b" alt="" width="375"><figcaption></figcaption></figure></div>

#### Gemma 4 12B

Google releases [Gemma 4 12B](https://unsloth.ai/docs/models/gemma-4), a new model that runs locally on 8GB RAM. [GGUF](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF) / [Guide](https://unsloth.ai/docs/models/gemma-4)

Gemma 4 12B Unified supports image, audio and 256K context. Run and train the model via Unsloth Studio.

#### MCP

* Remote `MCP` server support, including custom headers and OAuth
* Local command-based `MCP` server support
* `MCP` can now be turned on from the chat composer
* Built-in presets for common `MCP` servers

#### New Chat UI

* Projects, Canvas, `MCP`, RAG and Compare controls now live in the plus menu
* Search and Code controls are easier to access from the composer
* Menus, overlays, icons and clickable controls are more consistent across Unsloth

#### Projects

* Organize related chats into dedicated project workspaces
* Move existing chats into projects
* Create and manage projects directly from the sidebar

#### Experimental Canvas / Artifacts

* Opens generated HTML in a dedicated canvas panel inside Unsloth Studio
* Supports interactive outputs, including browser based visualizations and CDN-loaded packages
* Lets you switch between rendered preview and source code

#### Install, Runtime and Hardware

* Windows prebuilt installs no longer require the early `CUDA Toolkit` check
* Linux `llama.cpp` prebuilts now match the detected runtime `cudart` major
* `ROCm` gfx detection is forwarded into prebuilt selection
* `Blackwell`, `B300` and `ARM64` Linux support updates

To update Unsloth or install a new Unsloth Studio, you must use:

**macOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% hint style="warning" %}
**DO NOT USE `unsloth studio update` anymore since packaging will not get the latest updates!**
{% endhint %}
{% endupdate %}

{% update date="2026-05-31" tags="new-releases,v0.1.43-beta" %}

## CUDA 13.3, Windows, Mac

**To update Unsloth or install a new Unsloth Studio, you must use:**

**macOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% hint style="warning" %}
**DO NOT USE `unsloth studio update` anymore since packaging will not get the latest updates!**
{% endhint %}

#### Mac Updates

* Re-enabled `llama.cpp` prebuilt binaries for Apple Silicon (M1-M4) - Mac OS 14 / 15 / 26 (Tahoe)
* Apple Silicon Mac OS 13 (Ventura) is source build
* Intel (x86\_64) for Mac OS 13.3 / 14 / 15 / 26 (Tahoe) uses `llama.cpp` prebuilt binaries
* Intel for Max 13.0 - 13.2 is source build

#### Windows Updates

* CUDA 13.3 `llama.cpp` prebuilt binaries now work for Windows
* For CUDA 13.2, CUDA 13.1 and below, Windows devices uses CUDA 12.4 fallback - we'll work on CUDA 13.1 binaries soon.

#### CUDA 13.3 Update

* CUDA 13.3 non Linux binaries work. We'll still use CUDA 13.1 for now
* CUDA 13.3 solves the CUDA 13.2 gibberish problem - see <https://github.com/unslothai/unsloth/issues/4849>

#### Blackwell GPUs Update

* For now Blackwell will have delayed releases of `llama.cpp` prebuilt binaries sine CUDA 12.4 does not work - we are working to resolve this soon.
  {% endupdate %}

{% update date="2026-05-26" tags="new-releases,v0.1.42-beta" %}

## An update before Revamp.

Hey guys, we're doing one more-ish update before a major revamp which is likely coming this week or next week. Our revamp will change a lot of things, especially with new major features and a lot of design changes.

{% embed url="<https://github.com/user-attachments/assets/70456395-e016-4273-8256-35adb206267e>" %}

* NEW: [**API calling support**](https://unsloth.ai/docs/integrations/connections) now with image generation + editing, proper web search, code execution, auto prompt caching. Connect [OpenAI](https://unsloth.ai/docs/integrations/connections/openai), [Anthropic](https://unsloth.ai/docs/integrations/connections/anthropic-claude) and more.
* Proper support for **non-English languages** e.g. Japanese, Chinese, Indian etc.

Many of you may have missed our previous release which only lasted for one day. We introduced:

* Connect to external inference backends: [vLLM](https://unsloth.ai/docs/integrations/connections/vllm), [Ollama](https://unsloth.ai/docs/integrations/connections/ollama), [llama-server](https://unsloth.ai/docs/integrations/connections/connect-llama.cpp-to-unsloth-run-ggufs-with-llama-server)
* **Security improvements**
* **Auto MTP speculative decoding** for MTP GGUFs; get the best settings customized for your hardware.

#### API provider calling & external connections

* You can now connect Unsloth to any API cloud provider (OpenAI, Anthropic, OpenRouter etc.)
* **Built-in web search** for OpenAI, Anthropic, OpenRouter and Kimi
* **Built-in code execution** for OpenAI and Anthropic (Anthropic containers persist and are reused across turns)
* Prompt caching is enabled for OpenAI and Anthropic models saving 50 to 90% of costs.
* Image generation + editing
* API key now optional for local providers (llama.cpp / vLLM / Ollama)
* Auto-load models when adding a cloud provider

#### Other Unsloth Studio updates

* OpenDocument chat attachments
* o3 reasoning summary payload
* Sending/prompting non-English languages (e.g. Japanese, Chinese) now works properly
* IME composer hardening, RTL `dir="auto"`, long log-line truncation fix
* Tool reasoning trace rendering in UI
* Fully offline support: cached GGUF discovery and offline DNS auto-detect for both inference and training

#### Unsloth Studio security improvements

* Authentication rate-limiting, proxy-aware so reverse proxies don't bypass it
* Sandboxed worker with a tightened blocklist (bash, `hf upload`, `NOFILE`)
* Path containment so workers can't escape their in-flight tmp dirs
* Strict schema validation across the Unsloth API
* Tightened CSP / security headers (only legitimate favicon hosts allowed)
* Removed the `torch.load` fallback on `training_args.bin` so untrusted pickles can never execute on model load
* Hardened Tauri desktop release flow
* Frontend auth: singleflight token refresh, current-password input on changes, working logout, shared 422 helper
* Cancel cleanup now scoped strictly to in-flight tmp dirs so it can never delete user state
  {% endupdate %}

{% update date="2026-05-19" tags="new-releases,v0.1.41-beta" %}

## MTP + Unsloth Fixes

Lots of bug fixes, UI, UX fixes to Unsloth! To get the latest updates do:

**macOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

#### Fixes

1. Fix `unsloth studio update` not working well
2. Fix getting stuck on `reset-password` page
3. More offline mode support
4. Improve MTP not being faster on Macs, CPUs and GPUs - now it's much better!
5. Fix Desktop Shortcut not working after update
6. Many many UI UX bug fixes
   {% endupdate %}

{% update date="2026-05-18" tags="new-releases,model-release,v0.1.405-beta" %}

## Qwen3.6 MTP + API Connections

We've got lots of new updates for Unsloth `v0.1.41-beta`:

* **\~2x faster GGUF inference** with automatically enabled [MTP](/docs/models/qwen3.6#mtp-guide)
* [**API calling support**](/docs/integrations/connections) for [OpenAI](/docs/integrations/connections/openai), [Anthropic](/docs/integrations/connections/anthropic-claude) etc. with auto prompt caching, web search, code execution
* Connect to external inference backends: [vLLM](/docs/integrations/connections/vllm), [Ollama](/docs/integrations/connections/ollama), [llama-server](/docs/integrations/connections/connect-llama.cpp-to-unsloth-run-ggufs-with-llama-server)
* Experimental **MLX inference**
* Proper support for **non-English languages**
* **Security** improvements

<a href="/pages/NpuhjPsxi8BKhuS8nnyY#qwen3.6-inference-tutorials" class="button primary">Run Qwen3.6 Tutorials</a><a href="/docs/models/qwen3.6#mtp-guide" class="button primary">MTP Guide</a>

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F0abq31rgj0gOSH9vnpQ6%2Fmtp.gif?alt=media&amp;token=c66ec3f5-91f0-4617-824c-c537a2dde8b0" alt="" width="375"><figcaption></figcaption></figure></div>

#### MTP speculative decoding support 1.4 to 2x faster inference!

* **Auto MTP speculative decoding** for MTP GGUFs; warn when the bundled llama.cpp prebuilt is stale or too old for MTP
* New pre-built llama.cpp binaries for MTP support!

#### API provider calling & external connections

* You can now connect Unsloth to any API cloud provider (OpenAI, Anthropic, OpenRouter etc.)
* **Built-in web search** for OpenAI, Anthropic, OpenRouter and Kimi
* **Built-in code execution** for OpenAI and Anthropic (Anthropic containers persist and are reused across turns)
* Prompt caching is enabled for OpenAI and Anthropic models saving 50 to 90% of costs.
* API key now optional for local providers (llama.cpp / vLLM / Ollama)
* Auto-load models when adding a cloud provider

#### MLX inference (Experimental)

* MLX quants and models now can run locally on your Mac machines!
* We'll be adding thinking, tools and web search soon!

#### Other Unsloth Studio updates

* Sending/prompting non-English languages (e.g. Japanese, Chinese) now works properly
* OpenDocument chat attachments
* o3 reasoning summary payload
* IME composer hardening, RTL `dir="auto"`, long log-line truncation fix
* Tool reasoning trace rendering in UI
* Fully offline support: cached GGUF discovery and offline DNS auto-detect for both inference and training
* Lots of UI/UX polish: dark theme refactor, right sidebar redesign, time-of-day sloth mascot, dismissable copyable toasts, larger chat composer, code-execution config polish, composer action pill styling, narrower Discord button

#### Training updates

* Gemma attention mask fixes
* Multi Image GRPO
* GRPO hidden-state return experiments
* New Continued Pretraining (CPT) training method as a first-class option
* Gemma-4 MoE LoRA extractor registered to fix `grouped_mm` contraction crash
* Opt-in fused `lm_head` + cross-entropy forward, with single-matmul path under `UNSLOTH_RETURN_LOGITS=1`
* Pass batch size for eval
* Eval/training paths now honour `HF_DATASETS_OFFLINE` alongside `HF_HUB_OFFLINE`

#### Unsloth Studio security improvements

* Authentication rate-limiting, proxy-aware so reverse proxies don't bypass it
* Sandboxed worker with a tightened blocklist (bash, `hf upload`, `NOFILE`)
* Path containment so workers can't escape their in-flight tmp dirs
* Strict schema validation across the Unsloth API
* Tightened CSP / security headers (only legitimate favicon hosts allowed)
* Removed the `torch.load` fallback on `training_args.bin` so untrusted pickles can never execute on model load
* Hardened Tauri desktop release flow
* Frontend auth: singleflight token refresh, current-password input on changes, working logout, shared 422 helper
* Cancel cleanup now scoped strictly to in-flight tmp dirs so it can never delete user state
  {% endupdate %}

{% update date="2026-05-05" tags="new-releases,v0.1.39-beta,v0.1.38-beta" %}

## Unsloth API endpoint

#### ***v0.1.39-beta bug fix*** **May 5th 2026**

Fixes chat history not being shown (existing chat history is not lost) and attachments not attaching correctly. The bug was render-only - use `2026.5.2` or directly call `curl -fsSL https://unsloth.ai/install.sh | sh`  to update

You can use local LLMs with tools like [Claude Code](https://unsloth.ai/docs/basics/claude-code) and [Codex](https://unsloth.ai/docs/basics/codex) by connecting them to Unsloth’s API endpoint. This lets you run models like [Qwen](https://unsloth.ai/docs/models/qwen3.6) and [Gemma](https://unsloth.ai/docs/models/gemma-4) locally, with additional features such as self-healing tool calling, code execution, and web search.

Using Unsloth as an API inference endpoint is beneficial not only because it is easy to setup and fast, but also because Unsloth provides:

* [Self-healing tool calling](https://unsloth.ai/docs/new/studio/chat#auto-healing-tool-calling), which helps reduce broken or malformed tool calls by 50%
* [Code execution](https://unsloth.ai/docs/new/studio/chat#code-execution) support, allowing Bash and Python execution for more accurate code outputs.
* Advanced [Web search](https://unsloth.ai/docs/new/studio/chat#advanced-web-search) that visits and actually reads webpages to gather in-depth info.
* [Automatic inference settings](https://unsloth.ai/docs/new/studio/chat#auto-parameter-tuning) for GGUF models (temp, top-k etc.)

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1s98Id9xclzwMfxjXw2O%2Funsloth%20api%20cropped.png?alt=media&amp;token=64fac263-ca5b-4447-a740-41f58ec94904" alt="" width="375"><figcaption></figcaption></figure></div>

#### New models

We've also got a handful of new models to run including NVIDIA [Nemotron 3 Nano Omni](/docs/models/nemotron-3-nano-omni), IBM [Granite 4.1](/docs/models/ibm-granite-4.1) and [Mistral 3.5](/docs/models/mistral-3.5) Medium. We helped Mistral solve some issues with implementation in transformers and GGUFs.

#### Unsloth Updates

* Stopped Unsloth training runs can now resume from checkpoints.
* Chat threads now autosave and persist more reliably.
* DPO training hangs in multi-process setups were fixed.
* VLM GRPO support improved with MROPE updates.
* Unsloth’s stop button now properly stops generation.
* Fix chat template disappearing after browser refresh.
  {% endupdate %}

{% update date="2026-04-23" tags="new-releases,v0.1.37-beta" %}

## Brand New UI Redesign

Hey guys, we revamped the entire Unsloth Studio UI and UX experience to put an emphasis on chat and training:

* Added a collapsible sidebar based on community feedback

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FKoonE97b0H3RioMwVohd%2Fstudio%20new%20ui.gif?alt=media&amp;token=e37f2839-914e-48b8-8c81-2dac3ade9408" alt="" width="375"><figcaption></figcaption></figure></div>

* You can now delete chats and search past conversations

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fm6se3dOmBZH9d9Wlif4W%2FScreenshot%202026-04-23%20at%206.15.38%E2%80%AFAM.png?alt=media&amp;token=e83fe246-1f88-4bf7-aa15-baf7d3356676" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fa4EGTdaWQ6EA81PeC3JZ%2FScreenshot%202026-04-23%20at%206.01.36%E2%80%AFAM.png?alt=media&amp;token=1a1cc50d-e0e0-47cd-9c1a-80583a887794" alt=""><figcaption></figcaption></figure></div>

* New Preserve Thinking toggle for models that support it like Qwen3.6
* Cleaner, more consistent design with easier navigation
* Expanded Settings page with options to change your profile picture, name, and more

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FvMtRDH8cVXCu08OZ4TRf%2FScreenshot%202026-04-23%20at%206.18.35%E2%80%AFAM.png?alt=media&amp;token=d989c4b5-6293-402c-981c-3209b7e7d750" alt="" width="375"><figcaption></figcaption></figure></div>

* No more entering your Hugging Face token twice
* gpt-oss now has low, medium and high thinking toggles.
* Now uses latest llama.cpp prebuilt, even on Linux CUDA
* Lots of bug, consistency and stability fixes
* Kimi-K2.6 can now be run!
* We also added experimental API support. Guides, announcement etc will come next week.

Qwen3.6 was also also previously already supported in Unsloth Studio for running and training. You can train and run Qwen3.6-27B right now!
{% endupdate %}

{% update date="2026-04-22" tags="model-release,new-releases" %}

## **Qwen3.6-27B + Kimi K2.6**

[**Qwen3.6-27B**](/docs/models/qwen3.6) can now run (18GB RAM) and be fine-tuned in Unsloth Studio. Kimi K2.6 can also be ran in Unsloth (350GB RAM).

Unsloth Studio received many new updates so please update. Details and writeup coming in the next few days.
{% endupdate %}

{% update date="2026-04-16" tags="model-release,new-releases" %}

## **Qwen3.6**

[**Qwen3.6**](/docs/models/qwen3.6) can now run and be fine-tuned in Unsloth Studio. The model runs on 23GB RAM and is the strongest mid-sized LLM on nearly all benchmarks.
{% endupdate %}

{% update date="2026-04-11" tags="model-release" %}

## **Gemma 4 Update + MiniMax-M2.7**

[Gemma 4 GGUFs](https://huggingface.co/collections/unsloth/gemma-4) are now updated with Google's official chat template fixes (which fixed/improved tool-calling), along with the latest llama.cpp fixes. Update to the latest llama.cpp, re-download quants and you shouldn't see `unused token` issues anymore.\
\
[MiniMax-M2.7](/docs/models/tutorials/minimax-m27) is out now! You can run the model locally with our GGUFs in 4-bit quantization on 128GB RAM / unified memory. [**MiniMax-M2.7 GGUF**](https://huggingface.co/unsloth/MiniMax-M2.7-GGUF)
{% endupdate %}

{% update date="2026-04-08" tags="new-releases,v0.1.36-beta" %}

## **Gemma 4 Fixes**

We’ve updated Gemma 4 [with many fixes](/docs/models/gemma-4/train). These bugs are universal and affected all training packages and implementations and **did not originate from Unsloth**. We identified the bugs, fixed them, and Gemma 4 training now works properly in Unsloth.

You only need **8GB VRAM** to train **Gemma-4-E2B** locally. Unsloth trains Gemma 4 **\~1.5x faster while using \~60% less VRAM** than FA2 setups. For the full guide and notebooks on Gemma 4 training, [see our blog](/docs/models/gemma-4/train).

#### Gemma 4 Training Fixes

1. **Gradient accumulation** no longer causes loss explosions. Previously, losses could spike to **300–400**; expected loss is around **10–15**.
2. Fixed the **IndexError** affecting **26B** and **31B** inference in `transformers`.
3. Fixed gibberish outputs for **E2B/E4B** when `use_cache=False`. See [issue #45242](https://github.com/huggingface/transformers/issues/45242).
4. Fixed **float16 audio** overflow from `-1e9` values.

If you see losses above **13–15,** for example **100** or **300** - gradient accumulation is likely being handled incorrectly. This is fixed in both **Unsloth** and **Unsloth Studio**.

#### Gemma 4 Quant Re-uploads

We also updated our Gemma 4 GGUFs so you will need to re-download. Again, these quant issues are **not related to or caused by Unsloth**:

1. CUDA: check for buffer overlap before fusing - critical fix for `<unused24>` tokens - [PR #21566](https://github.com/ggml-org/llama.cpp/pull/21566)
2. `kv-cache`: support attention rotation for heterogeneous iSWA - [PR #21513](https://github.com/ggml-org/llama.cpp/pull/21513)
3. `vocab`: add byte token handling to BPE detokenizer for Gemma 4 - [PR #21488](https://github.com/ggml-org/llama.cpp/pull/21488)
4. `convert`: set `"add bos" == True` for Gemma 4 - [PR #21500](https://github.com/ggml-org/llama.cpp/pull/21500)
5. `common`: add Gemma 4 specialized parser - [PR #21418](https://github.com/ggml-org/llama.cpp/pull/21418)
6. `llama-model`: read `final_logit_softcapping` for Gemma 4 - [PR #21390](https://github.com/ggml-org/llama.cpp/pull/21390)
7. `llama`: add custom newline split for Gemma 4 - [PR #21406](https://github.com/ggml-org/llama.cpp/pull/21406)

#### Unsloth Studio Updates

* Add **speculative decoding** support (ngram-mod, on by default)
* Llama.cpp updated to use latest version with all Gemma 4 Fixes
* Fix Qwen3.5 and Gemma 4 training issues
* Enable exporting and saving of Gemma 4 models
* Harden sandbox security for terminal and python tools
* Let recipes use the model loaded in Chat
* Fix empty chat threads on navigation (and whenever switching tabs) and stabilize new chat flow
* Allow non-LLM recipes to run and move Data tab first in executions
* Reuse HF cached repo casing to prevent duplicate downloads
  {% endupdate %}

{% update date="2026-04-03" tags="new-releases,v0.1.36-beta" %}

## **Google - Gemma 4**

* You can now run and train the [Gemma 4](/docs/models/gemma-4) models in Unsloth.
* Intel Mac now works
* Pre-compiled binaries for llama.cpp for 2 Gemma-4 fixes:
  * vocab: fix Gemma4 tokenizer ([#21343](https://github.com/ggml-org/llama.cpp/pull/21343))
  * fix: gemma 4 template ([#21326](https://github.com/ggml-org/llama.cpp/pull/21326))
* Tool calls for smaller models are now more stable and don't cut off anymore
* Pre-compiled binaries for Windows, Linux, Mac, WSL devices - CPU and GPU
* Speculative Decoding added for non vision models (Gemma-4 is vision sadly and Qwen3.5)
* Context length is now properly applied.
* Web search now actually gets web content and not just summaries
* 90% reduced HF API calls - less rate limits
  {% endupdate %}

{% update date="2026-03-31" tags="new-releases,improvements" %}

## **+50% tool call accuracy + more support**

* Tool calls for all models are now **+30% to +80% more accurate.**
* Web search now actually gets web content and not just summaries
* Number of tool calls allowed are increased to 25 from 10
* Tool calls now terminate much better, so looping / repetitions will be reduced
* More **tool call healing** and de-duplication logic to stop tool callings from leaking XML as well
* Tested with `unsloth/Qwen3.5-4B-GGUF` (`UD-Q4_K_XL`), web search + code execution + thinking enabled.

| Metric                       | Before | After     |
| ---------------------------- | ------ | --------- |
| XML leaks in response        | 10/10  | 0/10      |
| URL fetches used             | 0      | 4/10 runs |
| Runs with correct song names | 0/10   | 2/10      |
| Avg tool calls               | 5.5    | 3.8       |
| Avg response time            | 12.3s  | 9.8s      |

#### New features

* Added **custom folders** so you can use any GGUFs in any folder - for now access in Advanced Settings in Chat and Custom Folders
* **Update button** now visible
* Install script styling all updated!
* Preliminary **Automatic Multi GPU support for inference and training** - useful for large models that don't fit on 1 GPU - Unsloth auto will allocate GPU resources
* Intel Macs should work out of the box

Much smoother and faster Unsloth

* **Fixed timeouts of downloads of large models** - no more timeouts seen.
* **Fixed Hugging Face rate limiting - HF API calls reduced by 90%**
* Fixed bun on Windows and faster installs
  {% endupdate %}

{% update date="2026-03-27" tags="new-releases,fixes,improvements" %}

## **New Important Updates**

It’s only been 2 days since our previous release, but we’ve got a more important updates:

* **Inference is now 20–30% faster.** Previously, tool-calling and repeat penalty could slow inference below normal speeds. Inference tokens/s should now perform the same as `llama-server` / `llama.cpp`.
* **Now Auto-detects older or pre-existing models** downloaded from **LM Studio, Hugging Face,** and similar sources.
* **Inference token/s speed is now calculated correctly.** Previously, tokens/s included startup time, which made the displayed speed look slower than it actually was. It should now reflect 'true' inference speed.
* **CPU usage no longer spikes.** Previously, inline querier identity changed every render, causing `useLiveQuery` to resubscribe continuously.
* **Unsloth Studio now has a shutdown x button and shuts down properly.** Previously, closing it after opening from the desktop icon would not close it properly. Now, launching from the shortcut also opens the terminal, and closing that terminal fully exits Unsloth Studio. If you still have it open from a previous session you can restart your computer or run `lsof -i :8888` then `kill -9 <PID>`.
* **Even better tool-calling and websearch** with reduced errors.
* Updated documentation with lots of new info on [deleting models, uninstalling](/docs/new/studio/install#uninstall) etc.
* **Cleaner, smarter install and setup logging across Windows and Linux.** Output is now easier to read with consistent formatting, quieter by default for a smoother experience, and supports richer `--verbose` diagnostics when you want full technical detail.
* You can now view your training history!
  {% endupdate %}

{% update date="2026-03-25" tags="new-releases,fixes,improvements" %}

## First Release post Unsloth Studio

Hey guys, this is our first release since we launched Unsloth Studio. Lots of new features and fixes:

* **You can now update Unsloth Studio!** Please update via the same install commands.
* **Windows** CPU or GPU now works seamlessly. Please reinstall!
* **App shortcuts**. Once installed, you can now launch in Windows, MacOS and Linux via a shortcut icon in the Start / Launch and Desktop.
* **Pre-compiled `llama.cpp` binaries** and `mamba_ssm` - 6x faster installs! Also <300MB in size for binaries.
* **50% reduced installation sizes** (-7GB or more savings), 2x faster installs and faster resolving. 50% smaller pypi sizes.
* **Tool calling improved.** Better llama.cpp parsing, no raw tool markup in chat, faster inference, a new Tool Outputs panel, timers.
* MacOS and CPU now have [Data Recipes](/docs/new/studio/data-recipe) enabled with multi-file uploading.
* **AMD support preliminary for Linux** only machines - auto detects.
* **Settings sidebar redesign.** Settings are now grouped into **Model, Sampling, Tools, and Preferences**
* **Context length** now adjustable. Keep in mind this is not needed as llama.cpp smartly uses the exact context you need via `--fit on`
* **Multi-file upload.** Data recipes now support multiple drag-and-drop uploads for PDF, DOCX, TXT, and MD, with backend extraction, saved uploads, and improved previews.
* **Colab** with free T4 GPUs with Unsloth Studio now fixed! [Try it here](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb). Due to pre-compiled binaries, it's also 20x faster!
* **Better chat observability.** Unsloth now shows `llama-server` timings and usage, a context-window usage bar, and richer source hover cards.
* **Better UX overall** - clickable links, better LaTeX parsing, tool / code / web tooltips for default cards and much more!
* **LiteLLM -** Unsloth Studio and Unsloth were **NOT** affected by the recent LiteLLM compromise. Nemo Data Designer used LiteLLM only up to `1.80`, not the affected `1.82.7` or `1.82.8`, and has since removed it entirely.
* We now have a new one line install command, just run:&#x20;

  <pre class="language-bash" data-overflow="wrap" data-expandable="true"><code class="lang-bash">curl -fsSL https://unsloth.ai/install.sh | sh
  </code></pre>

#### **Fixes:**

* **Windows/setup improvements.** Fixed silent Windows exits, Anaconda/conda-forge startup crashes, broken non-NVIDIA Windows installs, and missing early CUDA/stale-venv setup checks.
* **System prompts fixed.** They work again for non-GGUF text and vision inference.
* **Persistent system prompts and presets.** Custom system prompts and chat presets now persist across reloads and page changes.
* **GGUF export expanded.** Full fine-tunes, not just LoRA/PEFT, can now export to GGUF. Base model resolution is more reliable, and unsupported export options are disabled in the UI.
* **Chat scroll/layout fixes.** Fixed scroll-position issues during generation, thinking-panel layout shift, and viewport jumps when collapsing reasoning panels.
* **Smarter port conflict detection.** Unsloth now detects loopback conflicts, can identify the blocking process when possible, and gives clearer fallback-port messages.
  {% endupdate %}

{% update date="2026-03-17" tags="fixes,improvements" %}

## New Tool calling + Windows Stability

* Claude Artifacts works so HTML can be executed like a snake game inside the chat
* +30% more accurate tool calls esp for small models + Timer for tool calls
* Tool + Web Search outputs can be saved + Toggle auto healing tool on/off
* Many bug fixes - Windows CPU works, Mac more seamless, faster and smaller installs
  {% endupdate %}
  {% endupdates %}


# Introducing Unsloth Studio

Run and train AI models locally with Unsloth Studio.

We’re launching **Unsloth Studio** (Beta): an open-source, no-code web UI for training, running and exporting open models in one unified **local** interface.

<a href="/docs/new/studio#quickstart" class="button primary" data-icon="bolt">Quickstart</a><a href="/docs/new/studio#features" class="button secondary" data-icon="star">Features</a><a href="https://github.com/unslothai/unsloth" class="button secondary" data-icon="github">Github</a>

* **Run GGUF**, **MLX** and diffusion image/video models locally on **Mac**, Windows, Linux.
* Train 500+ models 2x faster with 70% less VRAM (no accuracy loss)
* Run and train text, diffusion vision, TTS audio, embedding models

{% hint style="success" icon="sparkles" %}
**NEW:** We're introducing our new [**Unsloth Desktop**](/docs/desktop) app! Download for [Mac](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation) and [Linux](/docs/get-started/install/linux).
{% endhint %}

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="image">Cover image</th></tr></thead><tbody><tr><td><h4>Accurate tool calls</h4></td><td>Get 50% more accurate, self-healing tool calls and sandboxed code execution</td><td><a href="/docs/new/studio/chat">Chat with models</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLL8RSnAuc7XdutjAVD4n%2Fcode%20exec.png?alt=media&amp;token=566a7385-69e0-46db-a022-bef8e2cad677">code exec.png</a></td></tr><tr><td><h4>Diffusion image/video</h4></td><td>Run and train image/video diffusion models like MiniMax H3 with 2× faster inference.</td><td><a href="/docs/basics/diffusion-image">Generate Images Diffusion</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHD8yIrQN6pS7qCNYMVxp%2Ffklrfklf.gif?alt=media&amp;token=9e7daea2-c9cc-404c-af53-325833224c94">fklrfklf.gif</a></td></tr><tr><td><h4>Make your agent local</h4></td><td>Connect local LLMs to agentic tools like Claude Code and Codex with model swapping</td><td><a href="/docs/integrations/unsloth-start">Unsloth Start</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FaXmr0nHDKoSkhX6ESjLz%2Fclaude%20code%20image.png?alt=media&amp;token=67e609e9-b329-4c19-9e4d-2aed0fe34f9c">claude code image.png</a></td></tr><tr><td><h4>Unlimited Web search</h4></td><td>Get unlimited, private and secure web search and deep research</td><td><a href="/docs/new/studio#web-search-upgraded">Unsloth Studio</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6hy8X5V7FVW0NSWJfQAj%2Fwebsearch%20nice.png?alt=media&amp;token=8a02b6ac-3c85-484f-9bc7-f7d3eee24cd9">websearch nice.png</a></td></tr><tr><td><h4>Run the latest models</h4></td><td>Discover, manage and train the latest Muse Glimmer, MiniMax-H3, Kimi K3, Qwen3.8 models.</td><td><a href="/docs/models/tutorials">Complete LLM Directory</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FTSjv63MOktgx6PbgHAi2%2Fmodel%20hub%20models.png?alt=media&amp;token=b884e283-ff8b-4d4c-b4cb-7864d4335aa0">model hub models.png</a></td></tr><tr><td><h4><strong>Deploy anywhere</strong></h4></td><td>Securely deploy models remotely and access Unsloth anywhere via Cloudflare HTTPS</td><td></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJDIIEQ8fFZy2g5snC2kp%2Fcloudflare%20url.png?alt=media&amp;token=aadb42b5-16f0-42d1-8059-c2611e7cd931">cloudflare url.png</a></td></tr></tbody></table>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHTlw8oO7wDIYyGqilty0%2Funsloth%20qwen3.8%20final.png?alt=media&amp;token=b8b32558-d0c8-4de8-bedf-7ad10e2c4c4a" alt=""><figcaption></figcaption></figure>

* **MacOS:** Training, MLX and GGUF inference all work inside of Unsloth.
* No dataset needed. [**Auto-create datasets**](/docs/new/studio/data-recipe) from **PDF, CSV, JSON, DOCX, TXT** files.
* Connect your local models to any agent including [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [Hermes](/docs/integrations/hermes-agent) and more.
* [Export or save](/docs/new/studio/export) your model to GGUF, 16-bit safetensor etc.
* [**Self-healing tool calling**](/docs/new/studio/chat#auto-healing-tool-calling) / advanced [**web search**](/docs/new/studio/chat#advanced-web-search) + [**code execution**](/docs/new/studio/chat#code-execution)
* [Auto inference settings](/docs/new/studio/chat#auto-parameter-tuning), edit chat templates, use Unsloth as an [**API endpoint**](#unsloth-as-an-api-endpoint).

{% hint style="success" %}
**For all the latest updates, see our** [**new changelog page here**](/docs/new/changelog)**!** ✨
{% endhint %}

## ⭐ Features

{% columns %}
{% column %}

### Execute code + heal Tool calling

Unsloth Studio lets LLMs run Bash and Python, not just JavaScript with bypass permissions. It also sandboxes programs like Claude Artifacts so models can test code, generate files, and verify answers with real computation.

E.g. Unsloth creates a sandbox to allow GLM-5.2 to execute code which shows as HTML preview in Canvas.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLL8RSnAuc7XdutjAVD4n%2Fcode%20exec.png?alt=media&amp;token=566a7385-69e0-46db-a022-bef8e2cad677" alt=""><figcaption><p>Accurate tool calls with sandboxed code execution</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### Web search upgraded

Unsloth's private, unlimited and secure web search actually visits pages directly to collect relevant information and data and doesn't just scan through website summaries. This provides outputs much more accurate / in-depth info and context.

E.g. Qwen3.5-4B searched 20+ websites and cited sources, with web search happening inside its thinking trace.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FY3nfB43TEk7k11zcE4wm%2Fthe%20big%20one.gif?alt=media&amp;token=335be087-7375-4f89-9039-71195ee44ab8" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### Unsloth as an API endpoint

You can now use local LLMs via tools like [Claude Code](/docs/basics/claude-code) and [Codex](/docs/basics/codex) by connecting it to [Unsloth's API endpoint](/docs/basics/api). This means you'll be able to directly run Qwen and Gemma models in those tools with Unsloth's inference which includes features like self-healing tool-calling, websearch etc.

You can also [connect a provider](/docs/integrations/connections) like OpenAI, Anthropic or vLLM to Unsloth.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1s98Id9xclzwMfxjXw2O%2Funsloth%20api%20cropped.png?alt=media&amp;token=64fac263-ca5b-4447-a740-41f58ec94904" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### **No-code training**

[Upload PDF, CSV, JSON](#data-recipes) docs, or YAML configs and start training instantly on NVIDIA. Unsloth’s kernels optimize LoRA, FP8, FFT, PT across 500+ text, vision, TTS/audio and embedding models.

Fine-tune the latest LLMs like [Qwen3.5](/docs/models/qwen3.5/fine-tune) and NVIDIA [Nemotron 3](/docs/models/nemotron-3). [Multi-GPU](/docs/basics/multi-gpu-training-with-unsloth) works automatically, with a new version coming.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRjAfHShyL7MfHfq6BStl%2Fonboarding%20updated.png?alt=media&amp;token=7cdde1a0-8f8c-4d25-9414-e28f35f211cd" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### **Run models locally**

[Search and run GGUF](/docs/new/studio/chat) and safetensor models with self-healing [tool calling](#execute-code--heal-tool-calling), advanced [web search](/docs/new/studio/chat#advanced-web-search), [auto inference](/docs/new/studio/chat#auto-parameter-tuning) settings, [**code execution**](/docs/new/studio/chat#code-execution) (Bash + Python), [APIs](/docs/basics/api). Upload images, docs, audio, code.

[Battle models side by side](https://unsloth.ai/docs/new/studio/chat#model-arena). Powered by llama.cpp + Hugging Face, Unsloth supports **multi-GPU inference,** automatic offloading and fitting and most models.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNOhWqx2nFNOSjiQlv3Bz%2Finklin.gif?alt=media&amp;token=2fdae9e3-e78b-4cd3-af07-a9c4989a937f" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### Data Recipes

[**Data Recipes**](/docs/new/studio/data-recipe) transforms your docs into useable / synthetic datasets via graph-node workflow. Upload unstructured or structured files like PDFs, CSV and JSON. Unsloth Data Recipes, powered by NVIDIA Nemo [Data Designer](https://github.com/NVIDIA-NeMo/DataDesigner), auto turns documents into your desired formats.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fcc9T0V8WsyjcuOE2sIVV%2Fdata%20recipes%20longer.png?alt=media&amp;token=5ae33e8d-09b1-45e0-8f5c-40dca8bbcf0c" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### Observability

Gain [complete visibility](/docs/new/studio/start#training-progress) into and control over your training runs. Track training loss, gradient norms, and GPU utilization in real time, and customize to your liking.

You can even view the training progress on other devices like your phone.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FCIrWHN1JzfaFNOoavmZS%2Fobserve%20new.png?alt=media&amp;token=21fdbc5b-a073-437a-b487-b5bdff4716f6" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### Export / Save models

[**Export any model**](/docs/new/studio/export), including your fine-tuned models, to safetensors, or GGUF for use with llama.cpp, vLLM, Ollama, LM Studio, and more.

Stores your training history, so you can revisit runs, export again and experiment.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F8UHzGTHF9q6LWrJy8Y4r%2FScreenshot%202026-03-15%20at%203.02.02%E2%80%AFAM.png?alt=media&amp;token=cb5e78f8-481a-4c9f-9361-db53e6e0ec37" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

### Privacy first + Secure

Unsloth Studio can be used 100% offline and locally on your computer. Its token-based authentication, including encrypted password and JWT access / refresh flows keeps your data secure.

You can use pre-exisiting / old models or GGUFs that previously downloaded from HF etc. Read [instructions here](/docs/new/studio/chat#using-old-existing-gguf-models).
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F15gRLbMDX1ReKdHBBl1G%2FScreenshot%202026-03-15%20at%203.54.51%E2%80%AFAM.png?alt=media&amp;token=ca096807-54c2-4d8c-bdc1-c1bb0055469b" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

## ⚡ Quickstart

### **Install with Unsloth Desktop**

The easiest way to install Unsloth Studio is with the native Desktop app. Download it for your operating system:

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

For a manual Unsloth Studio installation, use the commands below:

{% tabs %}
{% tab title="MacOS, Linux, WSL" %}
**Install Unsloth:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use the same command to update.
{% endtab %}

{% tab title="Windows" %}
**Install Unsloth:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

Use the same command to update.
{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
**Install Unsloth:**

```bash
brew install --cask unsloth
```

Update Unsloth:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
**Install Unsloth:**

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth:

<pre class="language-powershell"><code class="lang-powershell"><strong>winget upgrade --id UnslothAI.Unsloth -e
</strong></code></pre>

{% endtab %}
{% endtabs %}

#### Launch Unsloth

{% columns %}
{% column %}
Standard local launch:

```bash
unsloth studio
```

For more details about install and uninstallation please visit the [Unsloth Studio Install](/docs/new/studio/install) section.
{% endcolumn %}

{% column %}
Launch Unsloth securely over HTTPS:

```bash
unsloth studio --secure
```

Use a [free Cloudflare](/docs/basics/how-to-serve-local-llms-anywhere-secure-remote-access-with-cloudflare-and-unsloth) tunnel on Windows, Mac, or Linux.
{% endcolumn %}
{% endcolumns %}

{% content-ref url="/pages/XFZRr9F9hSOSIbG5lxqB" %}
[Installation](/docs/new/studio/install)
{% endcontent-ref %}

#### <i class="fa-google">:google:</i> Google Colab notebook

We’ve created a [free Google Colab notebook](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) so you can explore all of Unsloth’s features on Colab’s T4 GPUs. You can train and run most models up to 22B parameters, and switch to a larger GPU for bigger models. Just Click 'Run all' and the UI should pop up after installation.

{% columns %}
{% column %}
{% embed url="<https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb>" %}

Once installation is complete, scroll to **Start Unsloth Studio** and click **Open Unsloth Studio** in the white box shown on the left:

**Scroll further down, to see the actual UI.**
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkYitMrK55Ic6eIGqiKEJ%2FScreenshot%202026-03-16%20at%2011.21.16%E2%80%AFPM.png?alt=media&amp;token=4388c309-a598-41f3-9301-e434c334ac1c" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% hint style="warning" %}
Sometimes the Unsloth link may return an error. This happens because you might have disabled cookies or you're using an adblocker or Mozilla. You can still access the UI by scrolling below the button.
{% endhint %}

### 👾 Unsloth Start

{% columns %}
{% column width="58.333333333333336%" %}
[Unsloth Start](/docs/integrations/unsloth-start) lets you connect [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex) and other agents to local models via the `unsloth start` command.

Start Unsloth, load a model, open your project folder, and then run:

```bash
unsloth start claude
```

Replace `claude` with any agent below:
{% endcolumn %}

{% column width="41.666666666666664%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpXE6kCHjh8qOEaggf94M%2FScreenshot_20260718_122426.png?alt=media&amp;token=a59e4c8c-efdb-451b-b1f8-621955564f6d" alt="" width="563"><figcaption><p>Claude Code running with Qwen3.5 locally.</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

| Agent                                                              | Command                  |
| ------------------------------------------------------------------ | ------------------------ |
| <i class="fa-claude">:claude:</i> Claude Code                      | `unsloth start claude`   |
| <i class="fa-openai">:openai:</i> OpenAI Codex                     | `unsloth start codex`    |
| <i class="fa-caduceus">:caduceus:</i> Hermes Agent                 | `unsloth start hermes`   |
| <i class="fa-lobster">:lobster:</i> OpenClaw                       | `unsloth start openclaw` |
| <i class="fa-rectangle-vertical">:rectangle-vertical:</i> OpenCode | `unsloth start opencode` |

## <i class="fa-seedling">:seedling:</i> Workflow

Here is a usual workflow of Unsloth Studio to get you started:

1. Launch Unsloth from [install instructions](/docs/new/studio/install).
2. Load a model from local files or a supported integration.
3. Import training data from PDFs, CSVs, or JSONL files, or build a dataset from scratch.
4. Clean, refine, and expand your dataset in [Data Recipes](/docs/new/studio/data-recipe).
5. Start training with recommended presets or customize the config yourself.
6. Chat with the trained model and compare its outputs against the base model.
7. [Save or export](/docs/new/studio#export-save-models) locally to the stack you already use.

You can read our individual deep dives into each section of Unsloth Studio:

{% columns %}
{% column width="50%" %}
{% content-ref url="/pages/vrLQd9559vRkDY8zRR0h" %}
[Get Started](/docs/new/studio/start)
{% endcontent-ref %}

{% content-ref url="/pages/5ZU2kPF2eJ7VK0GeEUhu" %}
[Model Export](/docs/new/studio/export)
{% endcontent-ref %}
{% endcolumn %}

{% column width="50%" %}
{% content-ref url="/pages/m9k4PLFmjpsAP6LsQt7u" %}
[Data Recipes](/docs/new/studio/data-recipe)
{% endcontent-ref %}

{% content-ref url="/pages/FdMvLj95MbkAR4aHURvS" %}
[Chat with models](/docs/new/studio/chat)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

## <i class="fa-comments-question">:comments-question:</i> FAQ

**Does Unsloth collect or store data?**\
Unsloth does not collect usage telemetry. Unsloth only collects the minimal hardware information required for compatibility, such as GPU type and device (e.g. Mac). Unsloth Studio runs 100% offline and locally.

**How do I use an old / exisiting model that I downloaded previously from Hugging Face?**\
Yes, you can use pre-exisiting/old models or GGUFs that you previously downloaded from Hugging Face etc. They should be now be automatically detected by Unsloth otherwise read our [instructions here](/docs/new/studio/chat#using-old-existing-gguf-models).

**Why is inference sometimes slower in Unsloth?**\
Unsloth, like other local inference apps, are powered by llama.cpp, so speeds should be mostly the same. Sometimes Unsloth might be because you turned on web-search, code execution, self-healing tool-calling on. All these features may make your inference slower. If the speed difference is still slower with all features turned off, please make a GitHub issue!

**Does Unsloth Studio support OpenAI-compatible APIs?**\
Yes, see our [API endpoint guide here](/docs/basics/api).

**Is Unsloth now licensed under AGPL-3.0?**\
Unsloth uses a dual-licensing model of Apache 2.0 and AGPL-3.0. The core Unsloth package remains licensed under [**Apache 2.0**](https://github.com/unslothai/unsloth?tab=Apache-2.0-1-ov-file), while certain optional components, such as the Unsloth Studio UI are licensed [**AGPL-3.0**](https://github.com/unslothai/unsloth?tab=AGPL-3.0-2-ov-file).

This structure helps support ongoing Unsloth development while keeping the project open source and enabling the broader ecosystem to continue growing.

**Does Unsloth only support LLMs?**\
No. Unsloth supports a range of supported `transformers` compatible model families, including text, multimodal models, [text-to-speech](/docs/basics/text-to-speech-tts-fine-tuning), audio, [embeddings](/docs/basics/embedding-finetuning), and BERT-style models.

**Can I use my own training config?**\
Yes. Import a YAML config and Unsloth will pre-fill the relevant settings.

**Do you need to train models to use the UI?**\
No, you can just download any GGUF or model without fine-tuning any model.

#### Acknowledgements

A huge thank you to NVIDIA and Hugging Face for being part of our launch. Also thanks to all of our early beta testers for Unsloth Studio, we truly appreciate your time and feedback. We’d also like to thank llama.cpp, PyTorch and open model labs for providing the infrastructure that made Unsloth Studio possible.


# Get started with Unsloth Studio

A guide for getting started with the fine-tuning studio, data recipes, model exporting, and chat.

Unsloth Studio is a local, browser-based GUI for fine-tuning LLMs without writing any code. It wraps the training pipeline in a clean interface that handles model loading, dataset formatting, hyperparameter configuration, and live training monitoring.

<a href="https://unsloth.ai/docs/new/studio/start#unsloth-quickstart" class="button secondary" data-icon="bolt">Unsloth</a><a href="/docs/new/studio/start#data-recipes-quickstart" class="button secondary" data-icon="hat-chef">Data Recipe</a><a href="/docs/new/studio/start#export-quickstart" class="button secondary" data-icon="box-isometric">Export</a><a href="/docs/new/studio/start#chat-quickstart" class="button secondary" data-icon="comment-dots">Chat</a>

#### Setup Unsloth Studio

**Install with Unsloth Desktop.** The easiest way to install Unsloth Studio is with the native Desktop app. Download it for your operating system:

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

For a manual Unsloth Studio installation, use the commands below:

{% tabs %}
{% tab title="MacOS, Linux, WSL" %}
**Install Unsloth:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use the same command to update.
{% endtab %}

{% tab title="Windows" %}
**Install Unsloth:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

Use the same command to update.
{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}

First, launch Unsloth Studio using either a local install or a cloud option. Follow the [install instructions](/docs/new/studio/install) for your setup, or use our [free Colab](/docs/new/studio#google-colab-notebook) notebook. For a local setup, run:

```bash
unsloth studio
```

{% columns %}
{% column width="50%" %}
Open your browser of choice go to `http://127.0.0.1:8888`. You will need to create a new password.

**Launch Unsloth securely with HTTPS and Cloudflare.** Unsloth now provides a secure way to launch Unsloth over HTTPS through a free Cloudflare tunnel. Use the below (works in Windows, Mac & Linux):

```bash
unsloth studio --secure
```

{% endcolumn %}

{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZPtRrwafzmVX54HhyyBD%2FScreenshot%202026-03-16%20at%2011.25.22%E2%80%AFPM.png?alt=media&amp;token=9153c153-ec61-4fff-b1b9-db7f70ac2936" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

## <i class="fa-comment-dots">:comment-dots:</i> Chat - Quickstart

[Unsloth Studio Chat](/docs/new/studio/chat) lets you run models 100% offline on your computer. Run model formats like GGUF and safetensors from Hugging Face or from your local files.

* **Download + Run** any model like GGUFs, fine-tuned adapters, safetensors etc.
* [**Compare** different model](#model-arena) outputs side-by-side
* **Upload** documents, images, and audio in your prompts
* [**Tune** inference](#generation-settings) settings like: temperature, top-p, top-k and system prompt

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRCnTAZ6Uh88DIlU3g0Ij%2Fmainpage%20unsloth.png?alt=media&amp;token=837c96b6-bd09-4e81-bc76-fa50421e9bfb" alt="" width="563"><figcaption></figcaption></figure>

You can read our detailed tutorial / guide about running models with Unsloth Studio here:

{% content-ref url="/pages/FdMvLj95MbkAR4aHURvS" %}
[Chat with models](/docs/new/studio/chat)
{% endcontent-ref %}

### Model Loading Guide

Before using the API, you need to **load the model** you want to use in **Unsloth.** Open the **Select model** dropdown in the top-left corner of the chat page.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FuZqd6tcZ5LgMSh4ZND5x%2Fexport-1778505117710-24fps.gif?alt=media&amp;token=9defec95-5404-4654-9c33-67be967c9820" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
On a different page? Use the left sidebar and click `New Chat` to return to the chat page.
{% endhint %}

{% columns %}
{% column width="50%" %}

#### Select Model

Use the search bar to find the model you want to load into Unsloth.

Browse recommended models, search Hugging Face models directly, or set a custom model directory.

Locally trained and exported models can be loaded from the \`Fine-tuned\` tab.
{% endcolumn %}

{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FxZs1wCOyXbAeKVCjSMff%2Fimage.png?alt=media&amp;token=49812361-d723-4286-93c6-7df6989c33c5" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### GGUF Selection

Model repos contain multiple quantizations. Select the quantization most suitable for your available RAM / VRAM.\
\
In this guide we'll use `unsloth/gemma-4-26B-A4B-it-GGUF` and select the recommended `UD-Q4_K_XL` variant
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlNXz1I4EhHUFB5zihpSk%2Fimage.png?alt=media&amp;token=44733b70-9c03-4371-a836-9401b2748a0d" alt="" width="375"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

#### Downloading the model

{% columns %}
{% column %}
Search for the model you want to use, then **click it** to begin downloading and loading it.

After selecting a model variant, Unsloth will begin downloading and loading the model into memory.

Once loading is complete you will see the following confirmation:
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FzodWb7AaZH6HtE4lXopz%2Fimage.png?alt=media&amp;token=6f3ba3c1-54e2-4077-a1e7-11a9bebd820c" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsF5iWCQrYTBi6xcHxoUM%2Fimage.png?alt=media&amp;token=ed16b837-497a-48b5-9019-724f317f71dd" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

The model is loaded and ready to use. You can now chat with the model directly in Unsloth or connect it to tools such as [Claude Code](/docs/basics/claude-code) and [Codex](/docs/basics/codex).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FKoonE97b0H3RioMwVohd%2Fstudio%20new%20ui.gif?alt=media&amp;token=e37f2839-914e-48b8-8c81-2dac3ade9408" alt=""><figcaption></figcaption></figure>

## <i class="fa-bolt">:bolt:</i> Unsloth - Quickstart

Unsloth Studio homepage has 4 main areas: [Model](#id-1.-select-model-and-method), [Dataset](#id-2.-dataset), [Parameters](#id-3.-hyperparameters), and [Training/Config](#id-4.-training-and-config)

* **Easy setup for models and data** from Hugging Face or local files
* **Flexible training choices** like QLoRA, LoRA, or full fine-tuning, with defaults filled in
* **Helpful config tools** for splits, column mapping, hyperparameters and YAML configs
* **Great training visibility** with live progress, GPU stats, charts, startup status

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FxV1PO5DbF3ksB51nE2Tw%2Fmore%20cropped%20ui%20for%20homepage.png?alt=media&amp;token=f75942c9-3d8d-4b59-8ba2-1a4a38de1b86" alt="" width="563"><figcaption></figcaption></figure></div>

### 1. Select model and method

#### **Model Type**

Select the modality that matches your use-case:

| Type           | Use case                                |
| -------------- | --------------------------------------- |
| **Text**       | Chat, instruction following, completion |
| **Vision**     | Image + text (VLMs)                     |
| **Audio**      | Speech / audio understanding            |
| **Embeddings** | Sentence embeddings, retrieval          |

#### **Training Method**

Three methods are available, toggled with a pill selector:

| Method               | Description                               | VRAM    |
| -------------------- | ----------------------------------------- | ------- |
| **QLoRA**            | 4-bit quantized base model + LoRA adapter | Lowest  |
| **LoRA**             | Full-precision base model + LoRA adapter  | Medium  |
| **Full Fine-tuning** | All weights are trained                   | Highest |

Type any Hugging Face model name or search the Hub directly from the combobox. Local models stored in `~/.unsloth/studio/models` and your Hugging Face cache also appear in the list.

{% hint style="warning" %}
GGUF format models are excluded from training - they are inference only.
{% endhint %}

When you pick a model the Unsloth automatically fetches its configuration from the backend and pre-fills sensible defaults for all hyperparameters.

**HuggingFace Token**

Paste your Hugging Face access token here if the model is gated (e.g. Llama, Gemma). The token is validated in real-time and an error is shown inline if it is invalid.

### 2. Dataset

{% columns %}
{% column %}
Switch between two tabs to choose where your data comes from:

* **HuggingFace Hub** - live search against the Hub. The last-updated date is shown for each result.
* **Local** - drag-and-drop or click to upload a file unstructured or structured files like: `PDF`, `DOCX`, `JSONL`, `JSON`, `CSV`, or `Parquet` format. Previously uploaded datasets appear in a list that refreshes automatically.

You can view our detailed [Datasets Guide here](/docs/get-started/fine-tuning-llms-guide/datasets-guide).

Prompt Unsloth how to interpret and format your data:
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FCtWUm7GdHnKbe14fUQyT%2Fupdated_dataset.webp?alt=media&amp;token=3fcefe8d-b6a4-44c2-be9b-6dc282166095" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

| Format     | When to use                                 |
| ---------- | ------------------------------------------- |
| `auto`     | Let Unsloth detect the format automatically |
| `alpaca`   | `instruction` / `input` / `output` columns  |
| `chatml`   | OpenAI-style `messages` array               |
| `sharegpt` | ShareGPT-style conversations                |

**Splits and Slicing**

* **Subset** - automatically populated from the dataset card.
* **Train split / Eval split** - choose which splits to use. Setting an eval split enables the **Eval Loss** chart during training.
* **Dataset slice** - optionally restrict training to a row range (start index / end index) for quick experiments.

**Column Mapping**

If the Unsloth cannot automatically map your dataset columns to the correct roles a **Dataset Preview dialog** opens. It shows sample rows and lets you assign each column to `instruction`, `input`, `output`, `image`, etc. Suggested mappings are pre-filled where possible.

### 3. Hyperparameters

Parameters are grouped into collapsible sections. You can view our detailed [LoRA hyperparameters guide](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) here:

{% content-ref url="/pages/y6obKRSk8TwyjIrCjuGE" %}
[Hyperparameters Guide](/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide)
{% endcontent-ref %}

| Parameter          | Default | Notes                        |
| ------------------ | ------- | ---------------------------- |
| **Max Steps**      | `0`     | `0` means use Epochs instead |
| **Context Length** | `2048`  | Options: 512 → 32768         |
| **Learning Rate**  | `2e-4`  |                              |

**LoRA Settings**

*(Hidden when Full Fine-tuning is selected)*

| Parameter          | Default | Notes                                                                       |
| ------------------ | ------- | --------------------------------------------------------------------------- |
| **Rank**           | `16`    | Slider 4–128                                                                |
| **Alpha**          | `32`    | Slider 4–256                                                                |
| **Dropout**        | `0.05`  |                                                                             |
| **LoRA Variant**   | `LoRA`  | `LoRA` / `RS-LoRA` / `LoftQ`                                                |
| **Target Modules** | All on  | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |

For **Vision** models with an image dataset, four additional checkboxes appear. Fine-tune:

| Vision Layers | Language Layers | Attention Modules | MLP Modules |
| ------------- | --------------- | ----------------- | ----------- |

**Training Hyperparameters**

Organized into three tabs:

{% tabs %}
{% tab title="Optimization" %}

| Parameter             | Default     |
| --------------------- | ----------- |
| Epochs                | 3           |
| Batch Size            | 4           |
| Gradient Accumulation | 8           |
| Weight Decay          | 0.01        |
| Optimizer             | AdamW 8-bit |

{% endtab %}

{% tab title="Schedule" %}

| Parameter              | Default |
| ---------------------- | ------- |
| LR Scheduler           | linear  |
| Warmup Steps           | 5       |
| Gradient Checkpointing | unsloth |
| Random Seed            | 3407    |
| Save Steps             | 0       |
| Eval Steps             | 0       |
| Packing                | false   |
| Train on Completions   | false   |
| {% endtab %}           |         |

{% tab title="Logging" %}

| Parameter          | Default        |
| ------------------ | -------------- |
| Enable W\&B        | false          |
| W\&B Project       | llm-finetuning |
| Enable TensorBoard | false          |
| TensorBoard Dir    | runs           |
| Log Frequency      | 10             |
| {% endtab %}       |                |
| {% endtabs %}      |                |

{% hint style="info" %}
[**Unsloth Gradient Checkpointing**](/docs/blog/500k-context-length-fine-tuning#unsloth-gradient-checkpointing-enhancements)**: `unsloth`** uses Unsloth's custom memory-efficient implementation, which can reduce VRAM usage significantly compared to the standard PyTorch option. It is the recommended default.
{% endhint %}

### 4. Training and Config

The bottom-right card has three config management buttons and the **Start Training** button.

| Button     | Action                                        |
| ---------- | --------------------------------------------- |
| **Upload** | Load a previously saved `.yaml` config file   |
| **Save**   | Export the current config to YAML             |
| **Reset**  | Revert all parameters to the model's defaults |

The Start Training button stays disabled until a model and dataset are both configured. Validation errors appear inline - for example, setting eval steps without choosing an eval split, or pairing a text-only model with a vision dataset.

#### Loading Screen

{% columns %}
{% column %}
After you click **Start Training**, a full-page overlay appears while the backend prepares everything.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYtsUxHI0szGw8ZPxCHep%2Fimage.png?alt=media&amp;token=1701f4af-ef35-48da-80e7-4aba4e80f4d4" alt="" width="375"><figcaption></figcaption></figure></div>
{% endcolumn %}

{% column %}
The overlay shows an animated terminal with live phase updates:

* Blue: Downloading model / dataset
* Amber: Loading model / dataset
* Blue: Configuring
* Green: Training

You can cancel at any time using the **×** button in the corner. A confirmation dialog will appear before anything is stopped.
{% endcolumn %}
{% endcolumns %}

### Training Progress and Observability

Once the first training step arrives the overlay dismisses and the live training view is revealed. The fine-tuning process is complete when steps reach 100% on the progress bar. You can view the elapsed time and tokens.&#x20;

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fah3G1rYgRaDNY8Ay6Uw7%2Fimage.png?alt=media&amp;token=0528c15e-7a4b-4028-8070-95dc0871da5d" alt="" width="563"><figcaption></figcaption></figure></div>

{% columns %}
{% column %}

#### Status Panel

The left column shows:

* **Epoch** - current fractional epoch (e.g. `Epoch 1.23`)
* **Progress bar** - step-based, with percentage
* **Key metrics**:
  * **Loss** - training loss to 4 decimal places
  * **LR** - current learning rate in scientific notation
  * **Grad Norm** - gradient norm
  * **Model** - the model being trained
  * **Method** - `QLoRA` / `LoRA` / `Full`
* **Timing row** - elapsed time, ETA, steps per second, and total tokens processed
  {% endcolumn %}

{% column %}

#### GPU Monitor

The right column shows live GPU stats polled every few seconds:

* **Utilization** - percentage bar
* **Temperature** - °C bar
* **VRAM** - used / total GB
* **Power** - draw / limit in watts

#### Stopping Training

Use the **Stop Training** button in the top-right of the progress card. A dialog gives you two choices:

* **Stop & Save** - saves a checkpoint before stopping
* **Cancel** - stops immediately with no checkpoint
  {% endcolumn %}
  {% endcolumns %}

{% columns %}
{% column %}

#### Charts

Four live charts update as training progresses:

1. **Training Loss** - raw values plus an EMA-smoothed line and a running average reference line
2. **Learning Rate** - the LR schedule curve
3. **Gradient Norm** - gradient norm over steps
4. **Eval Loss** - only shown when you configured an eval split
   {% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRgXfe3sobdQWxha8yslr%2Fimage.png?alt=media&amp;token=b3aa9004-778b-4e3d-85b1-40a205ad0602" alt="" width="278"><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}
Each chart has settings (gear icon) with:

| Option             | Default             |
| ------------------ | ------------------- |
| Viewing window     | Last N steps slider |
| EMA Smoothing      | `0.6`               |
| Show Raw           | On                  |
| Show Smoothed      | On                  |
| Show Average line  | On                  |
| Scale (per series) | Linear / Log        |
| Outlier clipping   | No clip / p99 / p95 |
| {% endcolumn %}    |                     |

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFJtjQpAgOFaieyQCYhkq%2Fimage.png?alt=media&amp;token=4da9cdc2-c088-4ab8-8d0d-40d8d392ee03" alt="" width="276"><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

#### Config Files

{% columns %}
{% column %}
All training configurations can be saved and reloaded as YAML files. Files are named automatically as:

```
{model}_{method}_{dataset}_{timestamp}.yaml
```

{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FuGAKdGkANbh2wIENA9X7%2Fimage.png?alt=media&amp;token=9553db5b-5c88-4556-be49-fe61035edf11" alt="" width="178"><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

The YAML is structured into three sections:

{% code expandable="true" %}

```yaml
training:
  max_steps: 0
  num_train_epochs: 3
  per_device_train_batch_size: 4
  ...

lora:
  r: 16
  lora_alpha: 32
  ...

logging:
  report_to: none
  ...
```

{% endcode %}

This makes it easy to reproduce runs, share configurations, or version-control your experiments.

## <i class="fa-hat-chef">:hat-chef:</i> Data Recipes - Quickstart

[Unsloth Data Recipes](/docs/new/studio/data-recipe) lets you upload documents like PDFs or CSVs files and transforms them into useable datasets. Create and edit datasets visually via a graph-node workflow.

The recipes page is the main entry point. Recipes are stored locally in the browser, so you come back to saved work later. From here, you can create a blank recipe or open a guided learning recipe.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQ6e19jESrJg0VjHnX58c%2Fdata%20recipes%20final.png?alt=media&amp;token=8d74e453-815d-4790-83d1-76d0bc80a3ce" alt="" width="563"><figcaption></figcaption></figure></div>

Data Recipes follows the same basic path. You open the recipes page, create or pick a recipe, build the workflow in the editor, validate it run a preview, then run the full dataset once the output looks right. Add seed data and generation blocks, validate the workflow, preview sample output, then run a full dataset build. Unsloth Data Recipes is powered by NVIDIA [DataDesigner](https://github.com/NVIDIA-NeMo/DataDesigner).

At a glance a usual workflow should look like this:

1. Open the recipes page.
2. Create a new recipe or open an existing one.
3. Add blocks to define your dataset workflow.
4. Click **Validate** to catch configuration issues early.
5. Run a preview to inspect sample rows quickly.
6. Run a full dataset build when the recipe is ready.
7. Review progress and output live in graph or in **Executions** view for mode details.
8. Select the resulting dataset in **Unsloth** and fine tune a model.

## <i class="fa-box-isometric">:box-isometric:</i> Export - Quickstart

Use Unsloth Studio 'Export' to export, save, or convert models to GGUF, Safetensors, or LoRA for deployment, sharing, or local inference in Unsloth, llama.cpp, Ollama, vLLM, and more. Export a trained checkpoint or convert any existing model.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrrFY8YczW3dDpfYi1k9f%2FScreenshot%202026-03-15%20at%209.28.19%E2%80%AFPM.png?alt=media&amp;token=d2729e16-799f-48f0-8b07-0248b93fa599" alt="" width="563"><figcaption></figcaption></figure></div>

You can read our detailed tutorial / guide about exporting models with Unsloth Studio here:

{% content-ref url="/pages/5ZU2kPF2eJ7VK0GeEUhu" %}
[Model Export](/docs/new/studio/export)
{% endcontent-ref %}

## <i class="fa-video">:video:</i> Video Tutorial

{% hint style="warning" %}
The Unsloth Studio versions shown in the videos are old and are not reflective of the current version.
{% endhint %}

{% columns %}
{% column %}
{% embed url="<https://www.youtube.com/watch?v=mmbkP8NARH4>" %}

Here is a video tutorial created by NVIDIA to get you started with Unsloth:
{% endcolumn %}

{% column %}
{% embed url="<https://youtu.be/1lEDuRJWHh4?si=GHaS77ZZPOGjn3GJ>" %}

How to Install Unsloth Studio Video Tutorial
{% endcolumn %}
{% endcolumns %}

## Advanced Settings

### CLI Commands

The Unsloth CLI (`cli.py`) provides the following commands:

```
Usage: cli.py [COMMAND]

Commands:
  train             Fine-tune a model
  inference         Run inference on a trained model
  export            Export a trained adapter
  list-checkpoints  List saved checkpoints
  ui                Launch the Unsloth Studio web UI
  studio            Launch the studio (alias)
```

### Project Structure

{% code expandable="true" %}

```
new-ui-prototype/
├── cli.py                     # CLI entry point
├── cli/                       # Typer CLI commands
│   └── commands/
│       ├── train.py
│       ├── inference.py
│       ├── export.py
│       ├── ui.py
│       └── studio.py
├── setup.sh                   # Bootstrap script (Linux / WSL / Colab)
├── setup.ps1                  # Bootstrap script (Windows native)
├── setup.bat                  # Wrapper to launch setup.ps1 via double-click
├── install_python_stack.py    # Cross-platform Python dependency installer
└── studio/
    ├── backend/
    │   ├── main.py            # FastAPI app & middleware
    │   ├── run.py             # Server launcher (uvicorn)
    │   ├── auth/              # Auth storage & JWT logic
    │   ├── routes/            # API route handlers
    │   │   ├── training.py
    │   │   ├── models.py
    │   │   ├── inference.py
    │   │   ├── datasets.py
    │   │   └── auth.py
    │   ├── models/            # Pydantic request/response schemas
    │   ├── core/              # Training engine & config
    │   ├── utils/             # Hardware detection, helpers
    │   └── requirements.txt
    ├── frontend/
    │   ├── src/
    │   │   ├── features/      # Feature modules
    │   │   │   ├── auth/      # Login / signup flow
    │   │   │   ├── training/  # Training config & monitoring
    │   │   │   ├── studio/    # Main studio workspace
    │   │   │   ├── chat/      # Inference chat UI
    │   │   │   ├── export/    # Model export flow
    │   │   │   └── onboarding/# Onboarding wizard
    │   │   ├── components/    # Shared UI components (shadcn)
    │   │   ├── hooks/         # Custom React hooks
    │   │   ├── stores/        # Zustand state stores
    │   │   └── types/         # TypeScript type definitions
    │   ├── package.json
    │   └── vite.config.ts
    └── tests/                 # Backend test suite
```

{% endcode %}

### API Reference

All endpoints require a valid JWT `Authorization: Bearer <token>` header (except `/api/auth/*` and `/api/health`).

| Method | Endpoint              | Description                                        |
| ------ | --------------------- | -------------------------------------------------- |
| `GET`  | `/api/health`         | Health check                                       |
| `GET`  | `/api/system`         | System info (GPU, CPU, memory)                     |
| `POST` | `/api/auth/signup`    | Create account (requires setup token on first run) |
| `POST` | `/api/auth/login`     | Login and receive JWT tokens                       |
| `POST` | `/api/auth/refresh`   | Refresh an expired access token                    |
| `GET`  | `/api/auth/status`    | Check if auth is initialized                       |
| `POST` | `/api/train/start`    | Start a training job                               |
| `POST` | `/api/train/stop`     | Stop a running training job                        |
| `POST` | `/api/train/reset`    | Reset training state                               |
| `GET`  | `/api/train/status`   | Get current training status                        |
| `GET`  | `/api/train/metrics`  | Get training metrics (loss, LR, steps)             |
| `GET`  | `/api/train/stream`   | SSE stream of real-time training progress          |
| `GET`  | `/api/models/`        | List available models                              |
| `POST` | `/api/inference/chat` | Send a chat message for inference                  |
| `GET`  | `/api/datasets/`      | List / manage datasets                             |


# How to Run models with Unsloth

Run AI models, LLMs and GGUFs locally with Unsloth Studio.

[Unsloth Studio](/docs/new/studio) and Unsloth Desktop lets you run AI models 100% offline on your computer. Run model formats like GGUF and safetensors from Hugging Face or from your local files.

* **Works on all MacOS, CPU, Windows, Linux, WSL setups! No GPU required**
* [**Self-healing tool calling**](#auto-healing-tool-calling)**,** advanced [**web search**](#advanced-web-search), [**code execution**](#code-execution)
* Use Unsloth as an OpenAI-compatible inference [**API endpoint**](/docs/basics/api) or connect a [provider](/docs/integrations/connections)
* Search + Download + Run + [Compare](#model-arena) any model like GGUFs, LoRA adapters, safetensors etc.
* [**Auto inference parameter**](#auto-parameter-tuning) tuning (temp, top-p etc.) and edit chat templates
* Upload images, audio, PDFs, code, DOCX and more file types to chat with.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVrjsMhR9uLeIlqzD2jAL%2Fqwen37b.gif?alt=media&amp;token=f1bb11b4-d4fa-4765-83b7-bbba563082c9" alt="" width="563"><figcaption></figcaption></figure>

### Using Unsloth Studio Chat

{% hint style="success" %}
Unsloth Studio Chat automatically works on **multi-GPU setups** for inference.
{% endhint %}

{% columns %}
{% column %}

#### Code execution

Unsloth Studio lets LLMs run Bash and Python, not just JavaScript. It also sandboxes programs like Claude Artifacts so models can test code, generate files, and verify answers with real computation.

This makes answers from models more reliable and accurate.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fel6jjv4rUTRCRwcRpIr7%2Flong%20code%20exec.png?alt=media&amp;token=9d3d5930-0fdc-4d97-941c-983e5629296d" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Auto-healing tool calling

Unsloth Studio not only allows [tool calling](#id-50-tool-calling-accuracy), but also auto-fixes malformed or broken tool-calls by 50%.

This means you'll always get inference outputs **without** broken tool calling.&#x20;

E.g. Qwen3.5-4B searched 20+ websites and cited sources, with web search happening inside its thinking trace.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FXPQGEEr1YoKofrTatAKK%2Ftoolcallingif.gif?alt=media&amp;token=25d68698-fb13-4c46-99b2-d39fb025df08" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Advanced Web Search

Unsloth's unlimited and secure web search actually visits pages directly to collect relevant information and data and doesn't just scan through website summaries. This provides outputs much more accurate / in-depth info and context. Search uses DuckDuckGo's private and secure API.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVrjsMhR9uLeIlqzD2jAL%2Fqwen37b.gif?alt=media&amp;token=f1bb11b4-d4fa-4765-83b7-bbba563082c9" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Use Unsloth as an API endpoint

You can now use local LLMs via tools like [Claude Code](/docs/basics/claude-code) and [Codex](/docs/basics/codex) by connecting it to Unsloth's [API endpoint](#use-unsloth-as-an-api-endpoint). This means you'll be able to directly run Qwen and Gemma models in those tools with Unsloth's inference which includes features like self-healing tool-calling, websearch etc.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1s98Id9xclzwMfxjXw2O%2Funsloth%20api%20cropped.png?alt=media&amp;token=64fac263-ca5b-4447-a740-41f58ec94904" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Automatic inference settings

Inference parameters like **temperature**, **top-p**, **top-k**, [**MTP**](/docs/models/qwen3.6#mtp-guide) are automatically pre-set for new models like Qwen3.5 so you can get the best outputs without worrying about settings. You can also adjust parameters manually and edit the system prompt.

Context length adjustment is no longer necessary with llama.cpp’s smart auto context, which uses only the context you need without loading anything extra.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FAQKsjtynvCXKtadvKhq1%2FRecording%202026-03-13%20114257.gif?alt=media&amp;token=b5bfff0c-8189-4358-9344-08d0ae17782a" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Connect Providers

[Unsloth connects](/docs/integrations/connections) to OpenAI, Anthropic, Ollama, llama.cpp, vLLM, and others.

Add API keys or model server URLs, then use external models in the same chat interface as local + cloud models. Run with [prompt caching](/docs/integrations/connections#prompt-caching), tool-calling, thinking, and provider-native features like OpenAI's [web search](#web-search-and-thinking) and [code execution](#code-execution).
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FAQKsjtynvCXKtadvKhq1%2FRecording%202026-03-13%20114257.gif?alt=media&amp;token=b5bfff0c-8189-4358-9344-08d0ae17782a" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Search and run models

You can search and download any model via Hugging Face or use local files.

Unsloth supports a wide range of model types, including **GGUF**, vision-language, and text-to-speech models. Run the latest models like [Qwen3.5](/docs/models/qwen3.5) or NVIDIA [Nemotron 3](/docs/models/nemotron-3).

Upload images, audio, PDFs, code, DOCX and more file types to chat with.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBf3UDywdNSlvCBhUuVsp%2FScreenshot%202026-03-17%20at%2012.34.23%E2%80%AFAM.png?alt=media&amp;token=b6127cbf-76f7-48da-b869-3760ed5e9b42" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Chat Workspace

Enter prompts, attach any documents, images (webp, png), code files, txt, or audio as additional context, and see the model’s responses in real time.

Toggle on or off: Thinking + Web search.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHlOKWnSB6slhE1EXgAeZ%2Fimage.png?alt=media&amp;token=b5bdfe4e-fe0e-4a2a-9eba-b04b15a79018" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

### **+50% Tool Calling Accuracy**

Unsloth offers several unique features that improve tool calling, including:

* Tool calls across all models in Unsloth are **30% to 80% more accurate**.
* Web search retrieves actual web content instead of only summaries.
* The maximum number of allowed tool calls is **more than 25.**
* Tool calls terminate more reliably, reducing loops and repeated calls.
* Improved tool-call healing and deduplication logic helps prevent XML from leaking into outputs.

See test results with `unsloth/Qwen3.5-4B-GGUF (UD-Q4_K_XL)` with web search, code execution, and thinking enabled:

| Metric                       | Normal Tool-calling | Unsloth Tool-calling |
| ---------------------------- | ------------------- | -------------------- |
| XML leaks in response        | 10/10               | 0/10                 |
| URL fetches used             | 0                   | 4/10 runs            |
| Runs with correct song names | 0/10                | 2/10                 |
| Avg tool calls               | 5.5                 | 3.8                  |
| Avg response time            | 12.3s               | 9.8s                 |

### Model Arena

Unsloth Chat lets you compare any two models side-by-side using the same prompt. E.g. compare the base model and LoRa adapter. Inference will firstly load for one model, then the second one (parallel inference is being worked on).

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FC3xjqlunbpUr7nx6sQ4j%2Fimage.png?alt=media&amp;token=65501d63-1346-4a1e-b055-c94294a24305" alt="" width="563"><figcaption></figcaption></figure></div>

{% columns %}
{% column %}
After training, you can compare the base and fine-tuned models side by side with the same prompt to see what changed and whether results improved.

This workflow makes it easy to see how your fine-tuning changed the model’s responses and whether it improved results for your use case.
{% endcolumn %}

{% column %}

<div align="center" data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVgnE7eMPQk2vaFboJ4BU%2Fmodel%20arena%20closeup.png?alt=media&amp;token=8b0a910b-440c-4859-a846-0060e61e157b" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% hint style="success" %}
Unsloth Studio Chat auto works on **multi-GPU setups** for inference.
{% endhint %}

### Using old / existing GGUF models

{% columns %}
{% column %}
**Apr 1 update:** You can now select an existing folder for Unsloth to detect from.

**Mar 27 update:** Unsloth Studio now **automatically detects older / pre-existing models** downloaded from Hugging Face, LM Studio etc.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBn3Fs1cchFchl328wSOs%2FScreenshot%202026-04-05%20at%205.43.57%E2%80%AFAM.png?alt=media&amp;token=cc57ec6e-653a-4824-8e8d-a6bfbcd27493" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

**Manual instructions:** Unsloth Studio detects models downloaded to your Hugging Face Hub cache `(C:\Users{your_username}.cache\huggingface\hub)`. If you have GGUF models downloaded through LM Studio, note that these are stored in `C:\Users\{your_username}.cache\lm-studio\models` ***OR*** `C:\Users{your_username}\lm-studio\models` and are not visible to llama.cpp by default - you will need to move or copy those .gguf files into your Hugging Face Hub cache directory (or another path accessible to llama.cpp) for Unsloth Studio to load them.

After fine-tuning a model or adapter in Unsloth, you can export it to GGUF and run local inference with **llama.cpp** directly in Unsloth Chat. Unsloth Studio is powered by llama.cpp and Hugging Face.

### Adding Files as Context

Unsloth Chat supports multimodal inputs directly in the conversation. You can attach documents, images, or audio as additional context for a prompt.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FSitddQpGkOwUvirMem5P%2Fimage.png?alt=media&amp;token=43b7af91-ea86-4279-a787-b4b444640d82" alt="" width="563"><figcaption></figcaption></figure></div>

This makes it easy to test how a model handles real-world inputs such as PDFs, screenshots, or reference material. Files are processed locally and included as context for the model.

### **Deleting model files**

You can delete old model files either from the bin icon in model search or by removing the relevant cached model folder from the default Hugging Face cache directory. By default, Hugging Face uses `~/.cache/huggingface/hub/` on macOS/Linux/WSL and `C:\Users\<username>\.cache\huggingface\hub\` on Windows.

* **MacOS, Linux, WSL:** `~/.cache/huggingface/hub/`
* **Windows:** `%USERPROFILE%\.cache\huggingface\hub\`

If `HF_HUB_CACHE` or `HF_HOME` is set, use that location instead. On Linux and WSL, `XDG_CACHE_HOME` can also change the default cache root.

### **Unsloth not detecting or using my GPU**

If the model is not using your GPU specifically for Docker, try:

Pulling the latest image manually:

```bash
 docker pull unsloth/unsloth:latest
```

* Start the container with GPU access:
  * `docker run`: `--gpus all`
  * Docker Compose: `capabilities: [gpu]`
* On Linux, make sure the NVIDIA Container Toolkit is installed.
* On Windows:
  * Check that `nvcc --version` matches the CUDA version shown in `nvidia-smi`
  * Follow: <https://docs.docker.com/desktop/features/gpu/>


# Unsloth Studio Installation

Manually install Unsloth Studio, or download the separate Unsloth Desktop app.

Unsloth Studio works on Windows, Linux, WSL and MacOS. You should use the same installation process on every device, although the system requirements may differ by device.

### Install with Unsloth Desktop

The easiest way to install Unsloth Studio is with the native [Desktop app](/docs/desktop). Download it for your operating system:

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

The rest of this page installs **Unsloth Studio** manually.

### Manual Unsloth Studio installation

<a href="https://unsloth.ai/docs/new/studio/install#windows-1" class="button secondary" data-icon="windows">Windows</a><a href="https://unsloth.ai/docs/new/studio/install#macos-1" class="button secondary" data-icon="apple">MacOS</a><a href="https://unsloth.ai/docs/new/studio/install#linux-and-wsl" class="button secondary medium" data-icon="linux">Linux & WSL</a><a href="/docs/new/studio/install#docker" class="button secondary" data-icon="docker">Docker</a><a href="https://unsloth.ai/docs/new/studio/install#developer-nightly-installations-advanced" class="button secondary" data-icon="screwdriver-wrench">Developer Install</a>

### Quickstart

{% tabs %}
{% tab title="MacOS, Linux, WSL" %}
**Install Unsloth:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Use the same command to update.
{% endtab %}

{% tab title="Windows" %}
**Install Unsloth:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

Use the same command to update.
{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}

See below for more step-by-step instructions:

{% tabs %}
{% tab title="MacOS" %}

#### Step 1: Setup Unsloth

Launch the `terminal` from Mac, then install Unsloth by entering the command below.

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

or use **Homebrew**:

```bash
brew install --cask unsloth
```

Unsloth will start setting up the environment and installing the required packages as shown below. Type **Y** and Press `Enter` when asked if you want to allow Unsloth to start now. This will start Unsloth on your local **8888** port.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FnKjv9Qn86NpTc0jIAANL%2FStart_studio_windows_powershell.png?alt=media&amp;token=be7dfb36-80a9-4a5d-bcf6-063a887a471e" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="info" %}
If you chose not to start Unsloth during the installation process, you can always start the Unsloth app using `unsloth studio -p 8888` . If you would like to have your Unsloth instance accessible by clients outside of your PC/computer, add `-H 0.0.0.0` to the `unsloth studio` command.
{% endhint %}

#### Step 2: Start Unsloth

Open your browser of choice and go to the `http://127.0.0.1:8888` URL. If this is your first time installing Unsloth, you will be prompted to create a new password. After, the Unsloth app should now open on the Chat Page as shown below.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNX8VzAre3aWM86uUz9U6%2Fimage.png?alt=media&amp;token=7eccde46-6188-48a5-95de-05f0e8bf44be" alt="" width="375"><figcaption></figcaption></figure>

**Launch Unsloth securely with HTTPS and Cloudflare.** Unsloth now provides a secure way to launch Unsloth over HTTPS through a free Cloudflare tunnel. Use the below (works in Windows, Mac & Linux):

```bash
unsloth studio --secure
```

You can start training and running models immediately. You can view our more detailed step-by-step guide to get started below:

{% content-ref url="/pages/vrLQd9559vRkDY8zRR0h" %}
[Get Started](/docs/new/studio/start)
{% endcontent-ref %}
{% endtab %}

{% tab title="Windows" %}

#### Step 1: Setup Unsloth

Open the Start Menu, search for `PowerShell`, and launch it. Copy & paste the install command below:

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

it will begin installing automatically. After installation finishes, PowerShell will ask if you want to start Unsloth Studi&#x6F;**.**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FnKjv9Qn86NpTc0jIAANL%2FStart_studio_windows_powershell.png?alt=media&amp;token=be7dfb36-80a9-4a5d-bcf6-063a887a471e" alt="" width="375"><figcaption></figcaption></figure>

You can also launch it with the following command:

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

{% hint style="info" %}
If you would like to have your instance accessible by clients outside of your PC/computer.\
Add `-H 0.0.0.0` to the `unsloth studio` command.
{% endhint %}

#### Step 2: Start Unsloth

Open `http://127.0.0.1:8888` in your browser. On first launch, create a new password to continue to the Chat page. **Unsloth Studio** is now installed and ready to use.

The Unsloth app should now open on the Chat Page as shown below.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FtQZliTMxnBZcJSAidbOc%2Fimage.png?alt=media&amp;token=df1e9734-b7ba-448d-b600-1025035527ca" alt="" width="375"><figcaption></figcaption></figure>

**Launch Unsloth securely with HTTPS and Cloudflare.** Unsloth now provides a secure way to launch Unsloth over HTTPS through a free Cloudflare tunnel. Use the below (works in Windows, Mac & Linux):

```bash
unsloth studio --secure
```

You can start training and  running models immediately. You can view our more detailed step-by-step guide to get started below:

{% content-ref url="/pages/vrLQd9559vRkDY8zRR0h" %}
[Get Started](/docs/new/studio/start)
{% endcontent-ref %}
{% endtab %}

{% tab title="Linux, WSL" %}

#### Step 1: Setup Unsloth

{% tabs %}
{% tab title="Linux" %}
Open your terminal application. You can launch it by pressing `Ctrl + Alt + T`, or by searching for `Terminal` in your system's application menu.
{% endtab %}

{% tab title="WSL" %}
Click the Windows Start Menu, type the name of your installed distro (e.g. `Ubuntu`), then open it.

{% hint style="warning" %}
On **WSL**, make sure your **NVIDIA drivers** are installed on **Windows** (not inside WSL) and that the **CUDA toolkit** is installed inside your WSL distro. See the System Requirements below for details.
{% endhint %}
{% endtab %}
{% endtabs %}

To install, copy and run the install command:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Then:

1. Click inside the terminal window
2. Paste the command with `Ctrl + Shift + V`
3. Press `Enter`

Unsloth will start setting up the environment and installing the required packages as shown below. Type **Y** and Press `Enter` when asked if you want to allow Unsloth to start now. This will start Unsloth on your local **8888** port.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FaCUj9vIsSLtxrrRZ4UVx%2Fimage.png?alt=media&amp;token=27f413dd-65b2-4312-b344-506919bccc6c" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
If you chose not to start Unsloth during the installation process, you can always start the Unsloth app using `unsloth studio -p 8888` . If you would like to have your Unsloth instance accessible by clients outside of your PC/computer, add `-H 0.0.0.0` to the `unsloth studio` command.
{% endhint %}

#### Step 2: Start Unsloth

Open your browser of choice and type `http://127.0.0.1:8888` in the URL box. If this is your first time installing Unsloth, you will be forwarded to `http://127.0.0.1:8888/change-password` page as shown below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FnM9s9317lNmCQVJugLDU%2Fimage.png?alt=media&amp;token=740cb99d-23ab-4767-bfa2-41f49769d96d" alt="" width="563"><figcaption></figcaption></figure>

**Launch Unsloth securely with HTTPS and Cloudflare.** Unsloth now provides a secure way to launch Unsloth over HTTPS through a free Cloudflare tunnel. Use the below (works in Windows, Mac & Linux):

```bash
unsloth studio --secure
```

You can start training and  running models immediately. You can view our more detailed step-by-step guide to get started below:

{% content-ref url="/pages/vrLQd9559vRkDY8zRR0h" %}
[Get Started](/docs/new/studio/start)
{% endcontent-ref %}
{% endtab %}
{% endtabs %}

## System Requirements

### <i class="fa-windows">:windows:</i> Window**s**

Unsloth Studio works directly on Windows without WSL. To train models, make sure your system satisfies these requirements:

**Requirements**

* Windows 10 or Windows 11 (64-bit)
* NVIDIA GPU with drivers installed (for Intel GPUs, see [Intel GPUs](#intel-gpus))
* **App Installer** (includes `winget`): [here](https://learn.microsoft.com/en-us/windows/msix/app-installer/install-update-app-installer)
* **Git**: `winget install --id Git.Git -e --source winget`
* **Python**: version 3.11 up to, but not including, 3.14
* Work inside a Python environment such as **uv**, **venv**, or **conda/mamba**

### <i class="fa-apple">:apple:</i> MacOS

Unsloth Studio works on Mac devices for [Chat](#run-models-locally) and training and all features. You can run MLX models with Unsloth.

* macOS 12 Monterey or newer (Intel or Apple Silicon)
* Work inside a Python environment such as **uv**, **venv**, or **conda/mamba**

### <i class="fa-linux">:linux:</i> Linux & WSL

* Ubuntu 20.04+ or similar distro (64-bit)
* NVIDIA GPU with drivers installed (for Intel GPUs, see [Intel GPUs](#intel-gpus))
* CUDA toolkit (12.4+ recommended, 12.8+ for blackwell)
* Git: `sudo apt install git`
* Python: version 3.11 up to, but not including, 3.14
* Work inside a Python environment such as **uv**, **venv**, or **conda/mamba**

### <i class="fa-microchip">:microchip:</i> Intel GPUs

Install the latest Intel GPU driver first (see our [Intel guide](/docs/get-started/install/intel)). The installer then picks Intel XPU PyTorch by itself on Linux and Windows when an Intel Arc, Arc Pro, Core Ultra with Arc graphics, or Data Center GPU Max / Flex is the only GPU it finds. On Linux the installer's GPU line shows `Intel GPU (<device id>, XPU)` when this happens.

Pin XPU PyTorch yourself when the installer cannot see or pick the Intel GPU:

* **WSL**, where no PCI display device is visible to the installer
* a system that also has an NVIDIA or AMD GPU (one PyTorch install serves one vendor, so the other GPU will not be used for training)
* on Linux, `ONEAPI_DEVICE_SELECTOR`, `SYCL_DEVICE_FILTER` or `SYCL_DEVICE_ALLOWLIST` is set, or `ZE_AFFINITY_MASK` is set to anything other than empty or `default`

**Linux, WSL:**

{% code collapsedlinecount="10" %}

```bash
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_TORCH_INDEX_FAMILY=xpu sh

# or with the full index URL
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_TORCH_INDEX_URL=https://download.pytorch.org/whl/xpu sh
```

{% endcode %}

**Windows (PowerShell):**

{% code collapsedlinecount="10" %}

```powershell
$env:UNSLOTH_TORCH_INDEX_FAMILY = "xpu"; irm https://unsloth.ai/install.ps1 | iex
```

{% endcode %}

If you pinned XPU (for example on WSL), set the same variable every time you re-run the installer to update; without it the re-run installs CPU PyTorch again. To keep CPU PyTorch on a system with an Intel GPU, set `UNSLOTH_TORCH_INDEX_FAMILY=cpu` instead (or `UNSLOTH_DISABLE_XPU_AUTO=1` on Linux).

### <i class="fa-docker">:docker:</i> Docker

{% hint style="success" %}
Our Docker image now works for Unsloth! We're working on Mac compatibility.
{% endhint %}

* Pull our latest Unsloth container image: `docker pull unsloth/unsloth`
* Run the container via:

```bash
docker run -d -e JUPYTER_PASSWORD="mypassword" \
  -p 8888:8888 -p 8000:8000 -p 2222:22 \
  -v $(pwd)/work:/workspace/work \
  --gpus all \
  unsloth/unsloth
```

For more information, [see here](https://hub.docker.com/r/unsloth/unsloth#unsloth-docker-image).

* Access your studio instance at `http://localhost:8000` or external ip address `http://external_ip_address:8000/`

### <i class="fa-microchip">:microchip:</i> CPU only

Unsloth Studio supports CPU devices for [Chat](#run-models-locally) for GGUF models and [Data Recipes](/docs/new/studio/data-recipe)

* Same as the ones mentioned above for Linux (except for NVIDIA GPU drivers) and MacOS.

## Developer / Nightly Installations (Advanced)

### **Main Repo Install**

#### **macOS, Linux, WSL developer installs:**

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -p 8888
```

To install into an isolated location (its own virtual env, `auth/`, `studio.db`, cache and llama.cpp build), set `UNSLOTH_STUDIO_HOME` and pass it again at launch:

```bash
UNSLOTH_STUDIO_HOME="$PWD/.studio" ./install.sh --local
UNSLOTH_STUDIO_HOME="$PWD/.studio" unsloth studio -p 8888
```

Then to update :

```bash
cd unsloth && git pull
./install.sh --local
unsloth studio -p 8888
```

#### **Windows PowerShell developer installs:**

```powershell
git clone https://github.com/unslothai/unsloth.git
cd unsloth
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -p 8888
```

To install into an isolated location (its own virtual env, `auth/`, `studio.db`, cache and llama.cpp build), set `UNSLOTH_STUDIO_HOME` and pass it again at launch:

```powershell
$env:UNSLOTH_STUDIO_HOME="$PWD\.studio"; .\install.ps1 --local
$env:UNSLOTH_STUDIO_HOME="$PWD\.studio"; unsloth studio -p 8888
```

Then to update :

```powershell
cd unsloth; git pull
.\install.ps1 --local
unsloth studio -p 8888
```

#### **Remote access: `--secure` (HTTPS tunnel) vs raw port**

By default `unsloth studio` binds to `127.0.0.1` (this machine only). To reach it from another device, pick one of:

* `--secure` (recommended): serve **only** through a free Cloudflare HTTPS link. Unsloth stays bound to localhost and the tunnel provides the public URL; it fails closed (does not start) if the tunnel can't come up, so the raw port is never exposed.

```bash
unsloth studio --secure -p 8888
```

* `-H 0.0.0.0`: bind the raw port on all network interfaces, reachable from anywhere on the network. Only use this on a trusted network.

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

Server-side tools (web search, Python and terminal code execution) run as your user and are on by default. Anyone who can reach the server with the API key can run code on this machine, so keep your API key private and pass `--disable-tools` when exposing Unsloth.

### Advanced Launch methods

Installer options can be passed as environment variables. On macOS, Linux and WSL place the variable after the pipe so the shell passes it to `sh`; on Windows set it with `$env:` before piping to `iex`.

Skip PyTorch (GGUF-only mode):

```bash
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_NO_TORCH=1 sh
```

```powershell
$env:UNSLOTH_NO_TORCH=1; irm https://unsloth.ai/install.ps1 | iex
```

Pin the Python version:

```bash
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_PYTHON=3.12 sh
```

```powershell
$env:UNSLOTH_PYTHON='3.12'; irm https://unsloth.ai/install.ps1 | iex
```

Install to a custom location with `UNSLOTH_STUDIO_HOME`:

```bash
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_STUDIO_HOME=/abs/path sh
```

```powershell
$env:UNSLOTH_STUDIO_HOME='C:\path'; irm https://unsloth.ai/install.ps1 | iex
```

Cap Unsloth's native CPU thread pools on high-core hosts: `UNSLOTH_CPU_THREADS=8 unsloth studio -p 8888`.

### **Use an Old Version**

#### **Unsloth Desktop**

To use an old version of Unsloth Desktop, it is very simple. Just look through [our GitHub releases](https://github.com/unslothai/unsloth/releases) and select the specific release you want to install in the 'Assets' area.

#### **Unsloth Studio**

To use an old version of Unsloth Studio, follow the below with your desired version and names:

**macOS, Linux, WSL:**

```shellscript
uv pip install \
  --python "$HOME/.unsloth/studio/unsloth_studio/bin/python" \
  --reinstall --no-deps "unsloth==VERSION"

"$HOME/.unsloth/studio/unsloth_studio/bin/unsloth" studio -p 8888
```

**Windows:**

```shellscript
uv pip install `
  --python "$env:USERPROFILE\.unsloth\studio\unsloth_studio\Scripts\python.exe" `
  --reinstall --no-deps "unsloth==VERSION"

& "$env:USERPROFILE\.unsloth\studio\unsloth_studio\Scripts\unsloth.exe" studio -p 8888
```

### Uninstall

The recommended way to fully remove Unsloth Studio is the matching uninstall script for your OS. It stops any running servers, removes the install dir, the launcher data dir, the desktop shortcut, and any platform-specific entries (macOS `.app` bundle + Launch Services on Mac; Start Menu, `HKCU\Software\Unsloth` registry key and user `PATH` entries on Windows):

* ​ **MacOS, WSL, Linux:**

{% code overflow="wrap" expandable="true" %}

```bash
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
```

{% endcode %}

* ​ **Windows (PowerShell):**

{% code overflow="wrap" expandable="true" %}

```bash
 irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex
```

{% endcode %}

If you only want to drop the install dir and keep the launcher/shortcut for a later reinstall, you can instead run `rm -rf ~/.unsloth/studio` (Mac/Linux/WSL) or `Remove-Item -Recurse -Force "$HOME\.unsloth\studio"` (Windows). The model cache at `~/.cache/huggingface` is not touched by any of these.

#### Manual uninstall

If you prefer to remove only specific parts:

**1. Remove app only** (keeps history, chats, checkpoints, and exports intact):

* **macOS, WSL, Linux:** `rm -rf ~/.unsloth/studio/unsloth_studio`
* **Windows (PowerShell):** `Remove-Item -Recurse -Force "$HOME\.unsloth\studio\unsloth_studio"`

**2. Remove Unsloth entirely** (keeps other Unsloth tools intact):

* **macOS, WSL, Linux:** `rm -rf ~/.unsloth/studio`
* **Windows (PowerShell):** `Remove-Item -Recurse -Force "$HOME\.unsloth\studio"`

**3. Remove everything Unsloth-related:**

* **macOS, WSL, Linux:** `rm -rf ~/.unsloth`
* **Windows (PowerShell):** `Remove-Item -Recurse -Force "$HOME\.unsloth"`

{% hint style="warning" %}
Note: Step 3 deletes everything in history, chats, model checkpoints, and exports. This cannot be undone.
{% endhint %}

**4. Remove shortcuts and symlinks:**

**macOS:**

```shellscript
rm -rf ~/Applications/Unsloth\ Studio.app ~/Desktop/Unsloth\ Studio
```

**Linux:**&#x20;

```shellscript
rm -f ~/.local/share/applications/unsloth-studio.desktop ~/Desktop/unsloth-studio.desktop
```

**WSL / Windows (PowerShell):**

```shellscript
Remove-Item -Force "$HOME\Desktop\Unsloth Studio.lnk"
Remove-Item -Force "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk"
```

**5. Remove the CLI command:**

* **macOS, Linux, WSL:** `rm -f ~/.local/bin/unsloth`
* **Windows (PowerShell):** The installer added the venv's `Scripts` directory to your User PATH. To remove it, open Settings → System → About → Advanced system settings → Environment Variables, find `Path` under User variables, and remove the entry pointing to `.unsloth\studio\...\Scripts`.

{% hint style="info" %}
Note: Steps 1-5 dont touch your downloaded HF model files. See Deleting cached HF model files below if you want to reclaim that space.
{% endhint %}

### **Deleting cached HF model files**

You can delete old model files either from the bin icon in model search or by removing the relevant cached model folder from the default Hugging Face cache directory. By default, Hugging Face uses `~/.cache/huggingface/hub/` on macOS/Linux/WSL and `C:\Users\<username>\.cache\huggingface\hub\` on Windows.

* **MacOS, Linux, WSL:** `~/.cache/huggingface/hub/`
* **Windows:** `%USERPROFILE%\.cache\huggingface\hub\`

If `HF_HUB_CACHE` or `HF_HOME` is set, use that location instead. On Linux and WSL, `XDG_CACHE_HOME` can also change the default cache root.

### Using old / existing GGUF models

{% columns %}
{% column %}
**Apr 1 update:** You can now select an existing folder for Unsloth to detect from.

**Mar 27 update:** Unsloth Studio now **automatically detects older / pre-existing models** downloaded from Hugging Face, LM Studio etc.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBn3Fs1cchFchl328wSOs%2FScreenshot%202026-04-05%20at%205.43.57%E2%80%AFAM.png?alt=media&amp;token=cc57ec6e-653a-4824-8e8d-a6bfbcd27493" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

**Manual instructions:** Unsloth Studio detects models downloaded to your Hugging Face Hub cache `(C:\Users{your_username}.cache\huggingface\hub)`. If you have GGUF models downloaded through LM Studio, note that these are stored in `C:\Users{your_username}.cache\lm-studio\models` ***OR*** `C:\Users{your_username}\lm-studio\models` . Sometimes when they are not visible, you will need to move or copy those .gguf files into your Hugging Face Hub cache directory (or another path accessible to llama.cpp) for Unsloth Studio to load them.

After fine-tuning a model or adapter in Unsloth, you can export it to GGUF and run local inference with **llama.cpp** directly in Unsloth Chat. Unsloth Studio is powered by llama.cpp and Hugging Face.

### <i class="fa-google">:google:</i> Google Colab notebook

We’ve created a [free Google Colab notebook](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) so you can explore all of Unsloth’s features on Colab’s T4 GPUs. You can train and run most models up to 22B parameters, and switch to a larger GPU for bigger models. Just Click 'Run all' and the UI should pop up after installation.

{% columns %}
{% column %}
{% embed url="<https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb>" %}

Once installation is complete, scroll to **Start Unsloth Studio** and click **Open Unsloth Studio** in the white box shown on the left:

**Scroll further down, to see the actual UI.**
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkYitMrK55Ic6eIGqiKEJ%2FScreenshot%202026-03-16%20at%2011.21.16%E2%80%AFPM.png?alt=media&amp;token=4388c309-a598-41f3-9301-e434c334ac1c" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% hint style="warning" %}
Sometimes the Unsloth link may return an error. This happens because you might have disabled cookies or you're using an adblocker or Mozilla. You can still access the UI by scrolling below the button.

Google Colab also expects you to stay on the Colab page; if it detects inactivity, it may shut down the GPU session.
{% endhint %}

## Troubleshooting

<table><thead><tr><th width="211.5999755859375">Problem</th><th>Fix</th></tr></thead><tbody><tr><td>Python version error</td><td><code>sudo apt install python3.12 python3.12-venv</code> version 3.11 up to, but not including, 3.14</td></tr><tr><td><code>nvidia-smi not found</code></td><td>Install NVIDIA drivers from https://www.nvidia.com/Download/index.aspx</td></tr><tr><td><code>nvcc not found</code> (CUDA)</td><td><code>sudo apt install nvidia-cuda-toolkit</code> or add <code>/usr/local/cuda/bin</code> to PATH</td></tr><tr><td>llama-server build failed</td><td>Non-fatal, Unsloth still works, GGUF inference won't be available. Install <code>cmake</code> and re-run setup to fix.</td></tr><tr><td><code>cmake not found</code></td><td><code>sudo apt install cmake</code></td></tr><tr><td><code>git not found</code></td><td><code>sudo apt install git</code></td></tr><tr><td>Build failed</td><td>Delete <code>~/.unsloth/llama.cpp</code> and re-run setup</td></tr></tbody></table>


# Unsloth Data Recipes

Learn how to create, build and edit datasets with Unsloth Studio's Data Recipes.

Unsloth Studio's Data Recipes lets you upload documents like PDFs or CSVs files and transforms them into useable / synthetic datasets. Create and edit datasets visually via a graph-node workflow. This guide will get you started with the basics before you dive into Unsloth Data Recipes.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQ6e19jESrJg0VjHnX58c%2Fdata%20recipes%20final.png?alt=media&amp;token=8d74e453-815d-4790-83d1-76d0bc80a3ce" alt=""><figcaption></figcaption></figure></div>

### How Data Recipes works

Data Recipes follows the same basic path. You open the recipes page, create or pick a recipe, build the workflow in the editor, validate it run a preview, then run the full dataset once the output looks right. Add seed data and generation blocks, validate the workflow, preview sample output, then run a full dataset build. Unsloth Data Recipes is powered by **NVIDIA Nemo** [**Data Designer**](https://github.com/NVIDIA-NeMo/DataDesigner).

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fc5m3JX1kUA3UwmdcJcxH%2FArea.gif?alt=media&amp;token=33bbd908-7d6c-456a-bc58-ce495c0adca1" alt=""><figcaption><p>Example of generating dataset and fine-tuning a model</p></figcaption></figure></div>

At a glance a usual workflow should look like this:

1. Open the recipes page.
2. Create a new recipe or open an existing one.
3. Add blocks to define your dataset workflow.
4. Click **Validate** to catch configuration issues early.
5. Run a preview to inspect sample rows quickly.
6. Run a full dataset build when the recipe is ready.
7. Review progress and output live in graph or in **Executions** view for mode details.
8. Select the resulting dataset in **Unsloth** and fine tune a model.

### Get Started

The recipes page is the main entry point. Recipes are stored locally in the browser, so you come back to saved work later. From here, you can create a blank recipe or open a guided learning recipe.

{% hint style="info" %}
Recipes can be exported and imported, so it is easy to share workflows with other Unsloth users :tada:. If you are trying to build a specific dataset pattern, ask in Unsloth Discord. Someone may already have a recipe they can share.
{% endhint %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FAwKh7speKv3eXERFhPg2%2FScreenshot%202026-03-13%20at%2008.26.11.png?alt=media&amp;token=809ac83f-75c8-4ef2-9721-65971b3faaa5" alt="" width="563"><figcaption><p>Recipes landing page</p></figcaption></figure></div>

If you are new to concept of workflows, learning recipes are the fastest way to see how seed data, prompts, expressions, and validators fit together in one working example. If you already know the shape of dataset you want, starting empty is usually quicker.

#### Choose a starting path

<table><thead><tr><th>If you want to:</th><th>Start with:</th><th data-hidden></th></tr></thead><tbody><tr><td><sub><strong>Build a custom workflow quickly</strong></sub></td><td><sub><strong>Start Empty</strong></sub></td><td></td></tr><tr><td><sub><strong>Learn the product from an example</strong></sub></td><td><sub><strong>Start from Learning Recipe</strong></sub></td><td></td></tr><tr><td><sub><strong>Continue previous work</strong></sub></td><td><sub><strong>Open a saved recipe</strong></sub></td><td></td></tr></tbody></table>

### What you build in the editor

The editor is where the recipe takes shape. You add blocks from the block sheet, configure them in dialogs, connect them on the canvas, and then validate or run the workflow.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6A2uN85dkYAdP7v0eUyX%2Fworkflow.gif?alt=media&amp;token=b44d9682-1825-4ddb-88f6-485a2aea3359" alt="" width="563"><figcaption><p>Example of building product description workflow</p></figcaption></figure></div>

{% columns %}
{% column %}
The editor has a few core parts:

* The recipe header, where you rename the recipe and switch between **Editor** and **Executions**
* The canvas, where the recipe graph is shown
* The block sheet, where you add new blocks
* Configuration dialogs, where you define prompts, references, model aliases, validators and seed settings.
* The floating **Run** and **Validate** controls
* need to add more here

{% endcolumn %}

{% column %}
The most common blocks in reciper are:

* **Seed** for input data from hugginface, local structured files (or unstructured documents that get chunked into rows.
* **LLM + Models** for providers, model configs, LLM generation blocks, and shared tool profiles.
* **Expression** for jinja2-based transforms that do not require an LLM call.
* **Validators** for filtering bad generated code with built in linters for Python, SQL, and Javascript/Typescript.
* **Samplers** for deterministic columns such as categories and subcategories.
  {% endcolumn %}
  {% endcolumns %}

### How references work

Most blocks that produce data (with some exceptions) becomes a reference for later blocks. That is one of the main ideas behind Data Recipes. You create a value once, then reuse it in prompts, expressions, structured outputs, and validation steps.

{% hint style="info" %}
Jinja Expressions help you work with values that arleady exist in the recipe. You can reference nested fields like `{{customer.first_name}}` , join values like `{{customer.first_name}} {{customer.last_name}}` and add conditional logic with patterns such as `{% if condition %}...{% endif %}`&#x20;
{% endhint %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNJYWKeIDC6aqgT2dShsB%2FScreenshot%202026-03-13%20at%2010.06.14.png?alt=media&amp;token=9d20bb8c-8a25-4395-9616-6429021d76f0" alt="" width="563"><figcaption><p>Example of references shown in the editor</p></figcaption></figure></div>

For example:

* A category block named `domain` can be references as `{{ domain }}`
* a seed column can be used directly in an LLM prompt, the columns in your seed data (eg. HF dataset columns, csv)
* a structured LLM output can expose fileds for later prompts
* an expression block can combine earlyier values without another model call

### What happens after?

Preview runs are for quick iteration. They return sample rows and analysis in the editor so you can inspect the generated data before commiting to a full run.

Full runs create a persisted local dataset artifact. That output later appears in Unsloth's local dataset picker, where you can inspect it again and use it for fine-tuning. Optionally you can publish your dataset to you hugginface repo.

### Core building blocks

{% columns %}
{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FuphMi76re2aUX7JvFNce%2FScreenshot%202026-03-13%20at%2011-35-45%20Unsloth%20Studio.png?alt=media&amp;token=674eb5ef-5acb-4b32-ab85-11a2af2e210f" alt="" width="188"><figcaption><p>Core building blocks</p></figcaption></figure></div>
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsdoYKdurtGeu4YgVqu0q%2FScreenshot%202026-03-13%20at%2011-38-59%20Unsloth%20Studio.png?alt=media&amp;token=328c7ea6-591e-43fa-9e87-c71277a54736" alt="" width="188"><figcaption><p>Model and LLM blocks</p></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

#### Model setup is split into two usable layers:

* **Model provider** defines the endpoint and authentifcation
* **Model Config** defines the model name and inference settings

This setup works with hosted providers, self-hosted endpoints, `vLLM` , `llama.cpp` , or any OpenAI-compatible API that you run outside Unsloth.

{% hint style="info" %}
Recipes are not limited to one model. You can add multiple **Model providers** and **Model config** blocks, then use different models for different steps, such as one for coding and another for general text tasks.
{% endhint %}

After model setup, you can use Four LLM block types:

| Block          | Output            | Best for                                                    |
| -------------- | ----------------- | ----------------------------------------------------------- |
| LLM Text       | Free-form text    | Instructions, explanations, conversations, and descriptions |
| LLM Structured | JSON              | Output that need fixed fields and predictable structure     |
| LLM Code       | Code              | Python, SQL, Typescript and other code generation tasks     |
| LLM Judge      | Scored evaluation | Grading outputs with one or more user-defined score         |

#### Tool Profiles

{% columns %}
{% column %}
Tool profile blocks defines shared MCP based tool access for one or more LLM blocks. Use them when a generation step needs tools, such as looking up code documentation through `Context7`.

Image to the left shows Context7 MCP added and configured in Tool Profile block dialog:
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fu4GbfrjuQyiU7cN15fDY%2FScreenshot%202026-03-13%20at%2010.50.01.png?alt=media&amp;token=889a9425-bb40-4e41-9dab-5408b03bd3ca" alt="" width="375"><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

#### Validators

{% columns %}
{% column %}
Validor block primarly target LLM code block by running generated code outputs through Linter and syntax validation, this helps you keep bad or invalid code rows out of the final dataset by filtering them out. The built-in options cover Python, SQL, and JavaScript/TypeScript validation.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FobyvYzcfyJkloHfHMyVC%2FScreenshot%202026-03-13%20at%2011-39-08%20Unsloth%20Studio.png?alt=media&amp;token=19c8dbfa-c876-4f6e-9e62-5b8df515dc7b" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

### Validate, preview and run

Once the recipe workflow is in place, the next step is execution. The reccomended pattern is: validate first, preview for quick feedback and inspect the generated data in executions view, then run the full dataset when you feel the output satisfies your plan.

Use the execution controls in third order:

{% stepper %}
{% step %}

#### Validate

Click **Validate** to catch configuration issues.
{% endstep %}

{% step %}

#### Preview

Run a preview to inspect sample rows and analysis
{% endstep %}

{% step %}

#### Refine

Refine prompts, references, seed settings, or validators.

Iterate untill you feel satisfied with generated data
{% endstep %}

{% step %}

#### Run the full dataset build

{% endstep %}
{% endstepper %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFCcEaVt8xsaNsMFi2MoZ%2Funsloth%20chef.png?alt=media&amp;token=0266aa36-4ba7-4364-be59-5fe57936ef7f" alt="" width="188"><figcaption></figcaption></figure>


# Export models with Unsloth Studio

Learn how to export your safetensor or LoRA model files to GGUF or other formats.

Use [Unsloth Studio](/docs/new/studio) to export, save, or convert models to GGUF, Safetensors, or LoRA for deployment, sharing, or local inference in Unsloth, llama.cpp, Ollama, vLLM, and more. Export a trained checkpoint or convert any existing model.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrrFY8YczW3dDpfYi1k9f%2FScreenshot%202026-03-15%20at%209.28.19%E2%80%AFPM.png?alt=media&amp;token=d2729e16-799f-48f0-8b07-0248b93fa599" alt="" width="563"><figcaption></figcaption></figure></div>

{% stepper %}
{% step %}

### Select Training Run

Start by selecting the training run you want to export from. Each run represents a complete training session and may contain multiple checkpoints.

After choosing a run, select the checkpoint to export. A checkpoint is a saved version of the model created during training.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FzB12XFNP3UjoAT1l9vz3%2Fimage.png?alt=media&amp;token=021b8864-b2c5-4a92-927e-e23350610036" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

### Select Checkpoint

Later checkpoints typically represent the final trained model, but you can export any checkpoint depending on your needs.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F8VfRPUcY3w6zYfNmAIDn%2Fimage.png?alt=media&amp;token=42565a7d-e62f-4cf0-bd33-90422f1b2194" alt="" width="560"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

### Export Methods

Depending on your workflow, you can export a merged model, LoRA adapter weights, or a GGUF model for local inference.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fh4sPts9rJhHiGqf0UxIs%2Fimage.png?alt=media&amp;token=4f1d6a76-bd40-4471-ab8d-0b2fe33d0410" alt=""><figcaption></figcaption></figure></div>

Each export method produces a different version of the model depending on how you plan to run or share it. The table below explains what each option exports.

| Export Type      | Description                                                                                       |
| ---------------- | ------------------------------------------------------------------------------------------------- |
| Merged Model     | **16-bit model** with the LoRA adapter merged into the base weights.                              |
| LoRA Only        | Exports **only the adapter weights**. Requires the original base model.                           |
| GGUF / llama.cpp | Converts the model to **GGUF format** for Unsloth / llama.cpp **/** Ollama / LM Studio inference. |
| {% endstep %}    |                                                                                                   |

{% step %}

### Export / Save Locally

When exporting a model, you can choose where the resulting files should be saved. Models can be downloaded directly to your machine or pushed to the Hugging Face Hub for hosting and sharing.

Save the exported model files directly to your machine. This option is useful for running the model locally, distributing files manually, or integrating with local inference tools.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FfsBaE8V2o69jSyCVGIz4%2Fimage.png?alt=media&amp;token=4ef3fa06-d25b-424a-91e3-42debd3b6908" alt="" width="325"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

### Push to Hub

Upload the exported model to the Hugging Face Hub. This allows you to host, share, and deploy the model from a central repository.

You will need a Hugging Face write token to publish the model.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrvVnuVUYQWv2nkrgFxpK%2Fimage.png?alt=media&amp;token=5e0b91fe-5225-4bff-9fa9-ec1fb3867b1a" alt="" width="325"><figcaption></figcaption></figure></div>

{% hint style="success" %}
If you are already authenticated with the Hugging Face CLI, the write token can be left empty.
{% endhint %}
{% endstep %}
{% endstepper %}


# Security in Unsloth

Unsloth uses a range of security measures across Unsloth Studio and Unsloth Desktop to help protect/safeguard users. Key safeguards are outlined below.

### 1. Operating-system sandboxes

Unsloth uses **bubblewrap** on Linux, **Seatbelt** on macOS, and **MXC** on Windows. See [Sandboxing in Unsloth](/docs/new/studio/sandboxing-in-unsloth) for more details

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fopx5w9JIwYNWGi2VWD0S%2Fimage.png?alt=media&amp;token=8091a99d-fd15-47bf-89c3-45a5c558c660" alt=""><figcaption></figcaption></figure>

### 2. Hugging Face repository security scanners

All Hugging Face repositories are scanned. Reverse shells, cloud-metadata access, and credential theft are blocked outright. For example, `deepseek-ai/deepseek-ocr` and `moonshotai/Kimi-VL-A3B-Instruct` both need permission before running `trust_remote_code = True`:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FV42Li7ASTlRoaVBuCi8r%2Fsecurity-01-deepseek-remote-code.png?alt=media" alt="DeepSeek-OCR custom-code approval dialog with a high-severity exec/eval scanner flag"><figcaption><p>The upstream DeepSeek-OCR example asks for permission and shows an exec/eval finding in model code.</p></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FcjbNPE46ghs02GlCeawS%2Fsecurity-02-kimi-remote-code.png?alt=media" alt="Kimi-VL-A3B-Instruct custom-code approval dialog with a scanner flag"><figcaption><p>The Kimi-VL-A3B-Instruct example asks for permission and shows the scanner's advanced-obfuscation-pattern flag.</p></figcaption></figure>

We removed `eval` calls and other problematic code sections and provide `unsloth/DeepSeek-OCR` and `unsloth/DeepSeek-OCR-2`, for example:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fb01XsCClbvFsVel1FcOL%2Fsecurity-03-unsloth-deepseek.png?alt=media" alt="Unsloth DeepSeek-OCR custom-code approval dialog with no worrying files flagged"><figcaption><p>The adapted Unsloth DeepSeek-OCR repository still asks for custom-code permission, even when the scanner does not flag worrying files.</p></figcaption></figure>

### 3. Malware scanners

Unsloth checks Hugging Face's security-scan status for a model. For example, `mcpotato/42-eicar-street` is blocked from loading and shows why:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FjqiNuBgoc5OqUFfcfY0W%2Fsecurity-04-malware-block.png?alt=media" alt="Unsafe-files dialog blocking mcpotato/42-eicar-street and listing flagged files"><figcaption><p>The malware warning blocks the model and explains that the flagged files were never downloaded.</p></figcaption></figure>

### 4. Failed-login limits

Unsloth limits failed login attempts.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F3uHp3NutN8szExxV7lZr%2Fsecurity-05-login-limit.png?alt=media" alt="Unsloth sign-in screen showing a cooldown after too many failed login attempts"><figcaption><p>Failed-login rate limiting presents a retry countdown; this example shows 53 seconds remaining.</p></figcaption></figure>

### 5. HTTPS remote access

`unsloth studio --secure` serves an HTTPS-only endpoint through a Cloudflared tunnel.

The [remote-access guide](/docs/basics/how-to-serve-local-llms-anywhere-secure-remote-access-with-cloudflare-and-unsloth) distinguishes this from starting a tunnel on an already-running wildcard-bound server: the latter can leave the raw network port reachable.

The guide also warns that server-side tools run as your user, and that an API-key holder with network access can execute code on the machine. It recommends `--disable-tools` when exposing Unsloth. Keep access credentials private and review the full guide before enabling remote access. HTTPS encrypts the connection; it does not make privileged tool access harmless.

### 6. Passwords

Passwords use salted PBKDF2-HMAC-SHA256. The first-run administrator password is randomly generated, and the account must change it on first login.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FC5ncbwkNXCXRGsrKKtTm%2Fsecurity-06-password-login.png?alt=media" alt="Unsloth password sign-in screen"><figcaption><p>The password sign-in screen included in the security-practices document.</p></figcaption></figure>

### 7. Encrypted API keys

Saved provider credentials are encrypted. API keys typed into the browser are encrypted with a per-install RSA key.

### 8. Multi-account isolation

Managed accounts can only reach their own folders, have no access to the owner's Hugging Face token, need the owner's grant to use models, and are blocked from running repository code.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FjxPTtS99YYycwpYCK4ny%2Fsecurity-07-managed-accounts.png?alt=media" alt="Accounts settings showing an installation owner and separate private accounts"><figcaption><p>Accounts settings distinguish the installation owner from managed private accounts.</p></figcaption></figure>

### 9. Sandboxed artifacts

HTML and MCP artifacts render in sandboxed frames with their own Content Security Policy (CSP).

### 10. PyTorch model loading

The PyTorch 2.6+ minimum means that `.bin` weights load with `weights_only=True`.

### 11. Approval modes

Unsloth provides `ask`, `auto`, and `full` approval modes. The `auto` mode flags network and filesystem imports for approval, and file paths need approval. Dangerous shell commands are blocked.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F3QjAxThLgzRROgxtrFki%2Fsecurity-08-tool-approval.png?alt=media" alt="File-edit tool call waiting for approval with Allow, Always allow, and Deny controls"><figcaption><p>A file-edit tool call waits for approval, with Allow, Always allow, and Deny controls.</p></figcaption></figure>

### 12. npm minimum release age

The npm setting `min-release-age=7` refuses packages published in the last seven days.

### 13. Installation-script allowlist

An `allowScripts` list specifies the only packages allowed to run installation scripts. CI fails on anything unreviewed.

### 14. Codex Security and iterative reviews

Codex Security and iterative Codex reviews are used throughout development to catch security issues and bugs:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FgMgORmGSKJG6j1xUAxdO%2Fsecurity-09-codex-review.png?alt=media" alt="Codex Security review on a merged GitHub pull request"><figcaption><p>An example Codex Security review attached to a pull request. The displayed outcome applies to that reviewed commit.</p></figcaption></figure>

### 15. Lockfiles and reproducible npm installation

Lockfiles and `npm ci` are used everywhere, and the installer upgrades users to npm 11 or newer.

### 16. CI hardening

`lockfile_supply_chain_audit.py` refuses lockfiles that show signs of Shai-Hulud-style injection before any `npm ci` or `cargo fetch`.

`scan_npm_packages.py` and `scan_packages.py` scan npm tarballs and PyPI packages, including credential reads inside installation scripts.

Code linters guard against unsafe loaders and dynamic execution, with baselines. The compiled-code path also has a dynamic-execution allowlist.

### 17. Dependency audits

Unsloth uses `pip-audit`, `npm audit` including signature checks, and `cargo audit`.

### 18. Code analysis and dependency-update cooldowns

Unsloth uses **CodeQL** and **Semgrep**, alongside **Dependabot** with three- to seven-day cooldowns.

### 19. Pinned GitHub Actions

GitHub Actions are pinned to commit SHAs, with none on mutable tags.

### 20. llama.cpp release verification

Prebuilt llama.cpp binaries are checked against release SHA-256 digests. A separate workflow audits Windows llama.cpp signatures.

### 21. VirusTotal release scanning

All Desktop releases are scanned with VirusTotal.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FbAF1S0JxrvtmGrArUIwf%2Fsecurity-10-virustotal.png?alt=media" alt="VirusTotal scan result for Unsloth Studio Desktop showing zero detections out of 70 vendors"><figcaption><p>The supplied Desktop-release example shows 0 detections out of 70 vendors at the time of the scan.</p></figcaption></figure>

### Using these controls

Security checks reduce risk; they do not establish that a model, dependency, code snippet, or release artifact is safe in every situation. Review code-execution requests carefully, keep Unsloth and its dependencies up to date, and use the least permissive approval mode that meets your needs. Screenshots show specific examples; scan outcomes apply to the files or commits shown.


# Sandboxing in Unsloth

Unsloth uses bubblewrap on Linux, Seatbelt on macOS, and MXC on Windows.

Unsloth now has OS level sandboxing which makes all tool calls get isolated in some folders without doing OS level harm. Unsloth has 2 modes - **"low"** and **"high" sandbox security** - low employs Unsloth's software sandbox methods which are sophisticated string, AST and regex checks for all tool calls (disabling dangerous rm -rf, exfiltrate tokens etc). High is true OS level sandboxing, which we have for all platforms.

| Operating System                            | Sandbox Software           | Latency per call |
| ------------------------------------------- | -------------------------- | ---------------- |
| <i class="fa-windows">:windows:</i> Windows | **MXC** (Windows official) | 164ms            |
| <i class="fa-linux">:linux:</i> Linux       | Bubblewrap / Bwrap         | 50ms             |
| <i class="fa-apple">:apple:</i> Mac         | Seatbelt (Mac internal)    | 110ms            |

To enable OS level sandboxing:

* **Windows - MXC is pre-installed** for Windows 11 24H2 and higher.
* **Linux** - Bwrap will need to be installed - select "High" and we will install it for you
* **Mac - Seatbelt is pre-installed**, so Unsloth auto enables Seatbelt

To check if OS Sandboxing is Low or High, click on the permissions button:

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBdWF0MbLoJLwTeFVhX7j%2FScreenshot%202026-10-08%20at%207.00.50%E2%80%AFAM.png?alt=media&amp;token=a592513a-9f45-4e0b-80d4-b8a6456e37ac" alt="" width="563"><figcaption></figcaption></figure></div>

If you click "Learn More", you will head to the Sandbox Settings page, and you can see if you have software sandboxing enabled, and also how to install bwrap, MXC as well:

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FfDEvjiXwBf1Fmftsp28q%2FScreenshot%202026-10-08%20at%206.56.34%E2%80%AFAM.png?alt=media&amp;token=b2c99b4c-67ea-4c50-bb98-4a710e0eafee" alt="" width="563"><figcaption></figcaption></figure></div>

For all 3 operating systems, you can see how OS level sandboxing is enabled:

{% columns %}
{% column %}
Windows:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLnxxpIwGpSjut6R6kLn8%2FScreenshot%202026-10-08%20at%206.53.26%E2%80%AFAM.png?alt=media&amp;token=85be4503-5279-4716-a1d0-1efbb8126129" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
Mac:

<img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRNZyuJL7xkrXJwYxofto%2FScreenshot%202026-10-08%20at%206.51.19%E2%80%AFAM.png?alt=media&amp;token=04ee3ff8-209c-41f6-bed2-8c64173ea77f" alt="" width="375">
{% endcolumn %}

{% column %}
Linux:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FesfsF7INvYB1MpIQF3cn%2FScreenshot%202026-10-08%20at%206.53.20%E2%80%AFAM.png?alt=media&amp;token=33f9f7cd-df4e-4e74-89c4-0c482c64e3c5" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column width="50%" %}
You can see the sandbox in action where it blocks accesses outside of the workspace:

### Latency of sandboxing

We optimized software and hardware OS level sandboxing a lot, but OS level sandboxing definitely has a latency addon. See below for a table:
{% endcolumn %}

{% column width="50%" %}
![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FcyBycXfercRlu0iEttel%2Fnew%20sandbox.png?alt=media\&token=d9064a88-e5c8-4e1d-a261-e93f20739ea6)
{% endcolumn %}
{% endcolumns %}

| Operating System                            | Sandbox Software           | Low Sandbox | High Sandbox |
| ------------------------------------------- | -------------------------- | ----------- | ------------ |
| <i class="fa-windows">:windows:</i> Windows | **MXC** (Windows official) | 8ms         | 164ms        |
| <i class="fa-linux">:linux:</i> Linux       | Bubblewrap / Bwrap         | 3ms         | 50ms         |
| <i class="fa-apple">:apple:</i> Mac         | Seatbelt (Mac internal)    | 10ms        | 110ms        |

{% columns %}
{% column %}

### Disable Sandboxing / Bypass Permissions / Full Access

To disable sandboxing (both software low and OS level high), simply press "Full Access", and now Unsloth can now access your entire computer with 0 constraints.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6YpMXm0gOsM7lPaygu4T%2FScreenshot%202026-10-08%20at%206.45.21%E2%80%AFAM.png?alt=media&amp;token=b7f308f5-ac0b-4852-9eeb-961e6cc575ef" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### Software Sandboxing Approach

Software Sandboxing in Unsloth is a bunch of regular expressions, string checks and more - we add about **1-2ms of extra latency** per tool call, and we block the following:

* Deleting and disk tools: `rm, dd, mkfs, fdisk, mount, umount`
* Permissions and privilege: chmod, `chown, sudo, su, doas, pkexec, passwd`
* Network: `curl, wget, nc, ncat, netcat, socat, ssh, scp, sftp, rsync`
* Processes and power: `kill, killall, pkill, shutdown, reboot, halt, poweroff`
* Running another script's contents unseen: `eval, source`
* Windows only: `rmdir, takeown, icacls, runas, powershell, pwsh`

Python code, by reading the code before it runs:

* Shell escapes: `os.system`, and `subprked` command, or a non-literal command (for example pip install through su)
* Network calls: `requests, urllib`, raw socket connections
* Some sensitive system reads, such as `/etc/passwd`
* Tampering with signals or timeouts,used to dodge them.

Applied to every call in both modes:

* Secret environment variables are stripped.
* Studio's own credential files are refused
* Limits on processes, file size (100 MB), memory (8 GB) and CPU time (600 s), so a fork bomb hits the process limit, plus the call timeout

#### Windows MXC Partnership

Thank you to the Windows team for partnering with Unsloth on making MXC work well in Unsloth! [See here](https://blogs.windows.com/windowsexperience/2026/10/07/building-windows-for-hybrid-intelligence/) for the launch blog post

#### Getting sandboxing in Unsloth

Simply update Unsloth to the latest and you will get sandboxing!


# DeepSeek-V4.1-Flash: Run Locally Guide

Run DeepSeek-V4.1-Flash-1017 locally with Unsloth GGUFs and Engram n-gram offloading.

**DeepSeek-V4.1-Flash-1017** is the new update to V4.1-Flash, making it the **strongest** open model to date for coding, agentic tasks and more. With 1M context, its **552B** backbone + **196B Engram** parameters support SSD offloading to reduce RAM/VRAM needs. This guide shows how to run DeepSeek-V4.1 locally with [Unsloth Dynamic](/docs/basics/dynamic-3.0-ggufs) GGUFs.

Unsloth + llama.cpp enables \~**3x faster** DeepSeek-V4.1 inference than other CPU/VRAM/ngram repos. Our **250.4 GB** 1-bit `UD-IQ1_M` quant achieves **88.2% top-1 agreement** against original DeepSeek and runs locally on 128GB+ RAM setups. `UD-Q8_K_XL` (lossless) is **508.4 GB**, while `UD-Q4_K_XL` (near lossless) is **500.7 GB**. GGUF: [**DeepSeek-V4.1-Flash-1017-GGUF**](https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF)

<a href="/pages/P265CJSPfOXTtJ9qxh3j#run-deepseek-v4.1-flash-1017-tutorials" class="button primary">Run DeepSeek-V4.1 Tutorials</a><a href="/docs/models/deepseek-v4.1-flash#quantization-analysis" class="button secondary">Quant Analysis</a>

### ⚙️ Usage Guide

**DeepSeek-V4.1-Flash-1017** pairs MoE with **Engram conditional memory**: n-grams of up to four tokens access a few table rows per token, so the table need not stay in GPU memory. It's best to use Unsloth for inference as it enables up to **3x faster** DeepSeek-V4.1 inference than other CPU/VRAM/ngram-focused repos by optimizing ontop of llama.cpp.

1-bit `UD-IQ1_S` will work on 128GB RAM setups like a Macbook, DGX Spark or RTX Spark while 3-bit `UD-IQ3_XXS` will work on 256GB RAM setups like a Mac Studio.

Like the PLE / n-gram layers in our [Qwen3.8-Flash-Next](/docs/models/qwen3.8-next) guide, Engram’s 110.6–202.8 GB table can stay on SSD via mmap. Rows load on demand into the OS cache, saving RAM/VRAM while retaining the full disk requirement.

#### **DeepSeek-V4.1-Flash-1017 Requirements:**

**Table: Hardware requirements** (RAM + VRAM, or unified memory; Engram on SSD).

| 1-bit      | 2-bit      | 3-bit      | 4-bit  | 8-bit  |
| ---------- | ---------- | ---------- | ------ | ------ |
| 128–145 GB | 176–205 GB | 256–290 GB | 305 GB | 315 GB |

Estimates allow roughly 5–10 GB above non-Engram weights. Runtime and context can require more.

**n-gram storage:** Lookup formats and weight sizes appear below. Rest GB is the absolutely minimum amount of RAM/VRAM that is needed.

| Quant        | N-gram format / GB | Rest GB |
| ------------ | ------------------ | ------- |
| UD-IQ1\_S    | IQ4\_NL / 110.6    | 122.4   |
| UD-IQ1\_M    | IQ4\_NL / 110.6    | 139.8   |
| UD-IQ2\_XXS  | Q5\_K / 135.2      | 165.3   |
| UD-Q2\_K\_XL | Q5\_K / 135.2      | 199.6   |
| UD-IQ3\_XXS  | Q6\_K / 161.3      | 233.5   |
| UD-IQ3\_S    | Q6\_K / 161.3      | 262.3   |
| UD-Q3\_K\_XL | MXFP8 / 202.8      | 282.3   |
| UD-Q4\_K\_XL | MXFP8 / 202.8      | 297.9   |
| UD-Q8\_K\_XL | MXFP8 / 202.8      | 305.6   |

### Recommended Settings

DeepSeek recommends temperature = 1.0, top-p = 1.0 and disabling top-k and min-p:

| Parameter              | Setting            |
| ---------------------- | ------------------ |
| `temperature`          | `1.0`              |
| `top_p`                | `1.0`              |
| Maximum context window | `1,048,576` tokens |

DeepSeek recommends allowing **at least 256K output tokens** for its long-generation setup.

### 📊 Quantization Analysis

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZbUjhpI7wJTjf3pM1uwu%2Fsize_vs_top1(1).png?alt=media&amp;token=4e133008-8026-4e93-baa7-d7f2e2b81ecc" alt=""><figcaption></figcaption></figure>

Results compare against DeepSeek’s official reference on wiki1100 and long17k. Lower KL means closer next-token probabilities; higher top-1 means more matching predictions.

DeepSeek mixes **MXFP4**, FP8 and BF16. Q4 and Q8 retain **MXFP4** experts and the same Engram table; their labels do not describe every tensor’s precision. Q4 stores many non-expert weights in Q8\_0; Q8 uses BF16.

We also show when we change the engram precision from IQ4\_NL to Q5\_K to Q6\_K and MXFP8 and we provide **standalone engrams which allow you to mix n match.**

| Quant        | Total size | wiki1100 KL / top-1 | long17k KL / top-1 |
| ------------ | ---------- | ------------------- | ------------------ |
| UD-IQ1\_S    | 233.0 GB   | 0.457 / 82.8%       | 0.364 / 85.1%      |
| UD-IQ1\_M    | 250.4 GB   | 0.314 / 87.1%       | 0.241 / 88.2%      |
| UD-IQ2\_XXS  | 300.5 GB   | 0.203 / 90.5%       | 0.142 / 91.2%      |
| UD-Q2\_K\_XL | 334.8 GB   | 0.138 / 91.5%       | 0.0879 / 93.3%     |
| UD-IQ3\_XXS  | 394.7 GB   | 0.0625 / 94.6%      | 0.043 / 95.1%      |
| UD-IQ3\_S    | 423.6 GB   | 0.0316 / 96.0%      | 0.0195 / 96.8%     |
| UD-Q3\_K\_XL | 485.1 GB   | 0.0204 / 96.5%      | 0.0114 / 97.4%     |
| UD-Q4\_K\_XL | 500.7 GB   | 9.00e-03 / 98.2%    | 4.46e-03 / 98.5%   |
| UD-Q8\_K\_XL | 508.4 GB   | 4.76e-03 / 98.5%    | 2.81e-03 / 98.7%   |

**Choosing a quant:** IQ1\_S saves about 54% disk space versus Q8. IQ1\_M adds 17.4 GB and improves top-1 agreement by 4.3 points on wiki1100 and 3.1 on long17k. IQ3 variants reach 94.6–96.8% agreement.

#### **Thinking effort**

Thinking is enabled by default, with `reasoning_effort=75` (“high”). You can set an integer from **1 to 100**. Higher values ask the model to reason more thoroughly; they are not a token limit. The included template also accepts `low` = 50, `medium` = 62, `high` = 75 and `max` = 100.

For example, add one of these options to your launch command:

```bash
# Default thinking effort
--chat-template-kwargs '{"reasoning_effort":75}'

# Maximum thinking effort
--chat-template-kwargs '{"reasoning_effort":100}'

# Disable thinking
--chat-template-kwargs '{"enable_thinking":false}'
```

## Run **DeepSeek-V4.1-Flash-1017** Tutorials:

For this tutorial, we use `UD-Q2_K_XL`. Unsloth automatically selects the near optimal DeepSeek-V4.1 offloading, ngram etc. configurations ensuring your setup gets the near best speed possible.

### 🦥 Unsloth Guide

[Unsloth Desktop](https://unsloth.ai/docs/desktop) is our open-source UI for local AI on macOS, Windows and Linux. You can search and download GGUFs, configure CPU and GPU offloading, adjust inference settings and connect tools from one app.

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Search and download DeepSeek-V4.1-Flash-1017

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub, search for **DeepSeek-V4.1-Flash-1017** and select [unsloth/DeepSeek-V4.1-Flash-1017-GGUF](https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF). Download your desired quant using the sizes and memory requirements above.
{% endstep %}

{% step %}

#### Run DeepSeek-V4.1-Flash-1017

Thinking is enabled by default with `75`, you can change it in our toggle. Unsloth automatically selects the near optimal DeepSeek-V4.1 offloading, ngram etc. configurations ensuring your setup gets the near best speed possible.

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).

With UD-IQ1\_S, the 233.0 GB download includes a 110.6 GB Engram n-gram table and 122.4 GB of other weights. Use Engram on-demand loading to keep the lookup table on SSD, with additional RAM/VRAM for cached rows, context and runtime buffers. Follow the Usage Guide above for memory requirements.
{% endstep %}

{% step %}

#### Serve DeepSeek-V4.1-Flash-1017 with Unsloth API

You can use `unsloth run` command and serve DeepSeek-V4.1-Flash-1017 via an API using `llama-server` runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api) or [unsloth start](/docs/integrations/unsloth-start).

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/DeepSeek-V4.1-Flash-1017-GGUF:UD-Q2_K_XL \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0 \
    --chat-template-kwargs '{"reasoning_effort":75}'
```

{% endcode %}

Use a runtime with DeepSeek-V4.1-1017 and Engram on-demand loading support. Apply the SSD-offload and memory settings from the Usage Guide when keeping the n-gram table on disk.
{% endstep %}

{% step %}

#### Unsloth is now ready

You can connect DeepSeek-V4.1-Flash-1017 to your tools through Unsloth Desktop. Unsloth also supports the following workflows:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune other supported text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally

<div align="center"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYShiiub5gs5pCQLX6d4U%2FScreenshot%202026-08-15%20at%2012.18.24%E2%80%AFAM.png?alt=media&amp;token=d6996797-a144-4113-b4c3-155cca6c39a0" alt="Unsloth Desktop Agents settings interface example" width="563"><figcaption><p>Unsloth Desktop’s Agents settings (interface example). Select DeepSeek-V4.1-Flash-1017 and your downloaded quant when connecting a tool.</p></figcaption></figure></div>
{% endstep %}
{% endstepper %}

### 🦙 Run DeepSeek-V4.1-Flash in llama.cpp

{% stepper %}
{% step %}
We need to use our specific llama.cpp PR [here](https://github.com/ggml-org/llama.cpp/pull/30817). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev git -y
git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp fetch origin pull/30817/head:deepseek41
git -C llama.cpp checkout deepseek41
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
To run the model, you can do:

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF \
    --local-dir unsloth/DeepSeek-V4.1-Flash-1017-GGUF \
    --include "*UD-Q2_K_XL*"
```

Keep all seven shards in the same folder and point `--model` at the first shard. llama.cpp loads the remaining shards automatically.
{% endstep %}

{% step %}
Then to run it:

```bash
./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-V4.1-Flash-1017-GGUF/UD-Q2_K_XL/DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL-00001-of-00007.gguf \
    --temp 1.0 \
    --top-p 1.0 \
    --top-k 0 \
    --min-p 0.0 \
    --chat-template-kwargs '{"reasoning_effort":75}' \
    --load-mode mmap \
    --lazy-mode on \
    --override-tensor "per_layer_token_embd.weight=CPU"
```

This example loads the Engram n-gram table from SSD on demand. Keep `--load-mode mmap` and `--lazy-mode on` enabled for SSD offloading.
{% endstep %}
{% endstepper %}

### ⚡**DSpark** - Speculative Decoding

DeepSeek-V4.1 includes **DSpark**, a speculative decoding architecture that proposes multiple tokens before the main model verifies them. It is separate from the Engram n-gram lookup table: **Engram supplies learned memory; DSpark accelerates decoding**. The similarly named n-gram speculative decoding option is also a different feature.

For DSpark, pair the main model with a matching **V4.1-1017 drafter** and a runtime with DSpark support. Keep the drafter and main checkpoint versions matched, and reserve extra memory for the drafter. The CLI and server examples in this guide use ordinary decoding.

**DSpark enables inference for DeepSeek-V4.1-Flash-GGUF to be around 1.3 to 1.5x faster for decoding!**

We provide a Q8\_0 DSpark module at 8GB, and a full BF16 at 8.5GB.

### :checkered\_flag:Mix n matching the engram

You can also change the engram precision for DeepSeek-V4.1 to whatever you desire.

| Engram Precision | Size (GB) |
| ---------------- | --------- |
| MXFP8 (original) | 202.8 GB  |
| Q6\_K            | 161.3 GB  |
| Q5\_K            | 135.2 GB  |
| IQ4\_NL          | 110.6 GB  |

For example after downloading the dynamic 1-bit quant IQ1\_S which uses by default the IQ4\_NL engram, you can switch it out for the MXFP8 one by only downloading the MXFP8 engram at <https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF/tree/main/engram> called `DeepSeek-V4.1-Flash-1017-engram-MXFP8.gguf_file` To do this, follow the script below:

```bash
# 1. get the table + the script
hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF \
  --include "engram/*MXFP8*" "engram/set_split.py" --local-dir .

# 2. stamp it as shard 5 of 5 (UD-IQ1_S's table slot)
python engram/set_split.py engram/DeepSeek-V4.1-Flash-1017-engram-MXFP8.gguf_file 5 5

# 3. keep the old table aside, then rename the new one to the table shard's name
mv UD-IQ1_S/DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00005-of-00005.gguf UD-IQ1_S/engram-IQ4_NL.bak
mv engram/DeepSeek-V4.1-Flash-1017-engram-MXFP8.gguf_file \
   UD-IQ1_S/DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00005-of-00005.gguf

# 4. run as usual
llama-server -m UD-IQ1_S/DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf
```

## 📊 Benchmarks

### **GGUF KLD and Top-1% Benchmarks**

Our quants span 82.8–98.5% top-1 agreement against the official F32 reference on wiki1100 and 85.1–98.7% on long17k. UD-Q4\_K\_XL saves 7.7 GB compared with UD-Q8\_K\_XL, with an agreement gap of 0.3 percentage points on wiki1100 and 0.2 on long17k. Q8 also has lower KL divergence on both sets.

**We also alter the engram table precision from IQ4\_NL to Q5\_K, Q6\_K and MXFP8, and show how KLD gets affected**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FdIFvyiUQafZ2upn6VdhJ%2Fsize_vs_kld(1).png?alt=media&amp;token=e5d14eab-9143-4d0a-b1a6-f5427d995808" alt=""><figcaption></figcaption></figure>

### **Official Benchmarks**

DeepSeek reports the following results for **V4.1-Flash-1017** compared with the previous V4.1-Flash release. These are official model results, not measurements of the GGUFs in this guide. Source: [DeepSeek’s 1017 model card](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017).

| Benchmark                   | V4.1-Flash-1017 | V4.1-Flash   |
| --------------------------- | --------------- | ------------ |
| HLE without tools           | 42.6 (44.0†)    | 36.8 (39.1†) |
| Terminal-Bench 4.0          | 42.3            | 31.2         |
| DeepSWE v1.1                | 77.0            | 74.2         |
| ProgramBench                | 29.1            | 20.3         |
| NL2Repo-Bench               | 68.2            | 65.4         |
| Automation-Bench v1.0.6     | 60.5            | 54.8         |
| Agents’ Last Exam           | 35.4            | 31.8         |
| Apex Bench v1.1             | 72.1            | 64.6         |
| Chartography (with tools)   | 83.0            | 78.9         |
| BabyVision (with tools)     | 92.5            | 89.6         |
| ZeroBench-main (with tools) | 51.0            | 49.0         |

† Text-only subset of HLE. DeepSeek uses the Claude Code harness for Terminal-Bench 4.0 and Mini-SWE-V2 for DeepSWE v1.1. Scores depend on the evaluation setup, reasoning effort and context budget.


# Qwen3.8 - How to Run Locally

Guide to running Qwen3.8 quants including Qwen3.8-27B on your local setup.

Qwen3.8 is Qwen’s new model family, featuring Qwen3.8-**27B**, Qwen3.8-**2.4T-A95B** and Qwen3.8-**Max**. Qwen3.8-27B has **vision** and reasoning capabilities, a **256K context** window, and runs locally on **17GB RAM/VRAM** setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop). Qwen3.8-2.4T-A95B is a 2.4T parameter (95B active) model with rivaling GPT-5.6 Sol.

**Aug 19 Update:** Qwen3.8-27B GGUFs now use [Unsloth Dynamic V3.0](/docs/basics/dynamic-3.0-ggufs) for 10% more accuracy at the same size, largely outperforming others.

{% columns %}
{% column %} <a href="/pages/CLyZKmpoJZpdaJXhLW4v#run-qwen3.8-guide" class="button primary">Run Qwen3.8 Guide</a><a href="https://unsloth.ai/download" class="button primary">Download Unsloth</a>

Thank you Qwen for day zero access. Unsloth quants also include:

* **Developer Role Support** for agentic tools like Codex
* [MTP enabled](/docs/models/mtp) for fast inference
* **Tool calling:** Improved parsing nested objects to make tools succeed more

Full-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit [Unsloth](https://github.com/unslothai/unsloth) Dynamic GGUFs takes **397GB (91% smaller)**, and larger IQ1\_S takes 508GB.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6aw6IjO7lnYJhSyzXYwq%2Fvolcano%20qwen.gif?alt=media&amp;token=94e56117-d10d-45e8-82dc-3e134b2b9359" alt=""><figcaption><p>Dynamic 4-bit Qwen3.8-27B in Unsloth Desktop</p></figcaption></figure>

Unsloth quants:

* [Qwen3.8-**27B**-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF)
* [Qwen3.8-27B-**NVFP4**](https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4)
  {% endcolumn %}
  {% endcolumns %}

### :gear: Usage Guide

#### Qwen3.8-27B Requirements:

Qwen3.8-**27B** 4-bit quants work on 16-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM.\
**Table: Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

<table><thead><tr><th>1-bit</th><th>2-bit</th><th>3-bit</th><th>4-bit</th><th width="128">6-bit</th><th>8-bit</th><th>BF16</th></tr></thead><tbody><tr><td>7-8 GB</td><td>9-11 GB</td><td>12-14 GB</td><td>16-19 GB</td><td>23-26 GB</td><td>31 GB</td><td>56 GB</td></tr></tbody></table>

{% hint style="info" %}
If you want to use [MTP](/docs/models/mtp) for faster inference, prepare to have 1-2GB extra headroom.
{% endhint %}

#### Qwen3.8-**2.4T** Requirements:

* [Qwen3.8-**2.4T-A95B**-GGUF](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF)

| Dynamic 1-bit XXXS | Dynamic 1-bit Standard | Dynamic 2-bit | Q8\_0  | BF16 (Lossless) |
| ------------------ | ---------------------- | ------------- | ------ | --------------- |
| 397GB              | 508GB                  | 657 GB        | 2.6 TB | 4.9 TB          |

### Recommended Settings

#### Qwen3.8-**27B Settings:**

Qwen3.8-27B is a **hybrid thinking** model with different default settings for thinking and non-thinking modes. Extra high is enabled by default so if you want shorter thinking traces, you can [adjust the thinking effort](#thinking--preserve-thinking):

| Parameter            | Thinking Mode | Instruct (non-thinking) Mode |
| -------------------- | ------------- | ---------------------------- |
| `temperature`        | 1.0           | 0.7                          |
| `top_p`              | 0.95          | 0.80                         |
| `top_k`              | 20            | 20                           |
| `min_p`              | 0.0           | 0.0                          |
| `presence_penalty`   | 0.0           | 1.5                          |
| `repetition_penalty` | 1.0           | 1.0                          |

* **Maximum context window:** `262,144` (can be extended to 1M via YaRN)
* Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`
* Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`

#### Qwen3.8-**2.4T Settings:**

Qwen3.8-2.4T is **thinking-only**, while Qwen3.8-Max is hybrid.

| Default                 |
| ----------------------- |
| temperature = 1.0       |
| top\_p = 0.95           |
| top\_k = 20             |
| min\_p = 0.0            |
| presence\_penalty = 0.0 |

* Context length = up to `1,010,000`
* `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`

If the model fits, you will get \~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading.

### 💡 Thinking + Preserve Thinking

{% columns %}
{% column %}
Qwen3.8 has **Preserve Thinking** which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. [Unsloth](#run-qwen3.8-in-unsloth-desktop) has 'Think' and Preserved Thinking toggles for Qwen3.8 (see right):
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLdgmjRrb5qhpbY9PwYe8%2FScreenshot%202026-08-14%20at%2011.26.15%E2%80%AFAM.png?alt=media&amp;token=6333f5ca-196d-46ae-9efd-2e522014e6db" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

Qwen3.8-27B comes with support for `reasoning_effort`, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:

* `xhigh` (default): for complex tasks demanding thorough analysis
* `medium`: balancing accuracy and speed
* `low`: efficient reasoning optimizing for speed and cost
* none

{% hint style="warning" %}
To change[ thinking / reasoning](#how-to-enable-or-disable-reasoning-and-thinking) effort in `unsloth run` or `llama-server`, use `--chat-template-kwargs '{"reasoning_effort":"medium"}'`

If you're on **Windows** Powershell, use: `--chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"`

Change `medium` to your desired reasoning level.
{% endhint %}

## Run Qwen3.8 Guide

You can now run Qwen3.8 in llama.cpp and Unsloth Desktop. For the large Qwen3.8-2.T model, we will be utilizing the 397GB `IQ1_XXXS` quant (named Q1\_0) for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type.

* Hugging Face: [Qwen3.8-27B-**GGUF**](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) • [Qwen3.8-27B-**NVFP4**](https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4)
* ModelScope: [Qwen3.8-27B-**GGUF**](https://www.modelscope.cn/models/unsloth/Qwen3.8-27B-GGUF) • [Qwen3.8-27B-**NVFP4**](https://www.modelscope.cn/models/unsloth/Qwen3.8-27B-NVFP4)
* **2.4T-A95B:** [Qwen3.8-**2.4T-A95B**-GGUF](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF)

<a href="/pages/CLyZKmpoJZpdaJXhLW4v#run-qwen3.8-in-unsloth-desktop" class="button primary">Run in Unsloth Desktop</a><a href="/pages/CLyZKmpoJZpdaJXhLW4v#run-qwen3.8-in-llama.cpp" class="button secondary">Run in llama.cpp</a><a href="/pages/CLyZKmpoJZpdaJXhLW4v#run-qwen3.8-in-llama.cpp" class="button secondary">NVFP4 Guide</a>

### 🦥 Run Qwen3.8 in Unsloth Desktop

Qwen3.8 can run in [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally), MLX and safetensor models
* [**Self-healing** tool calling](/docs/new/studio/chat#auto-healing-tool-calling) + **web search**
* [**Code execution**](/docs/desktop#code-execution) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via MLX and llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6IXaXdTVyvbrnjehlxys%2Fkimik3.gif?alt=media&amp;token=31e1213b-d7da-46e9-bc7f-3a8c402513fc" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Search and download Qwen3.8

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fo53hIyoFypinWWLa1c0x%2FScreenshot%202026-08-14%20at%2012.42.38%E2%80%AFPM.png?alt=media&amp;token=a69de985-56ee-4ef3-8e10-f669cc168346" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run Qwen3.8

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).

For example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fb3IiYJzHzsp2698xTim4%2Fgiffyy%20gf.gif?alt=media&amp;token=f1bda1a1-b81f-43e2-ba10-dbdc9a29c0c0" alt="" width="563"><figcaption><p>Dynamic 1-bit 397GB 91% smaller GGUF of Qwen3.8 2.4T in Unsloth Desktop</p></figcaption></figure>
{% endstep %}

{% step %}

#### Serve Qwen3.8 with Unsloth API

You can use `unsloth run` command and serve Qwen3.8 via an API using `llama-server` runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api) or [unsloth start](/docs/integrations/unsloth-start).

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --reasoning-effort medium
```

{% endcode %}
{% endstep %}

{% step %}

#### Unsloth is now ready

You can also do many other things with Qwen3.8 via Unsloth Desktop like:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYShiiub5gs5pCQLX6d4U%2FScreenshot%202026-08-15%20at%2012.18.24%E2%80%AFAM.png?alt=media&amp;token=d6996797-a144-4113-b4c3-155cca6c39a0" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Qwen3.8-2.4T-A95B New 1-bit data-types

We extended IQ1\_S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization (PTQ) without the need for QAT or QAD (quantization aware training / distillation). [Qwen3.8-**2.4T-A95B**-GGUF](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF)

Due to naming issues, we used TQ2\_0, TQ1\_0 and Q1\_0 otherwise it won't pop up in the HF repo.

<table><thead><tr><th>Dtype</th><th width="147.60000610351562">Naming</th><th width="114.39996337890625" align="right">BPW</th><th width="106.20001220703125" align="right"># entries</th><th width="113.413330078125" align="right">Index bits</th><th width="106.4000244140625" align="right">Block</th></tr></thead><tbody><tr><td>IQ1_S</td><td>IQ1_S</td><td align="right"><strong>1.5625</strong></td><td align="right">2048</td><td align="right">11</td><td align="right">50 B</td></tr><tr><td>UD-IQ1_XS</td><td>TQ2_0</td><td align="right">1.4375</td><td align="right">1024</td><td align="right">10</td><td align="right">46 B</td></tr><tr><td>UD-IQ1_XXS</td><td>TQ1_0</td><td align="right">1.3125</td><td align="right">512</td><td align="right">9</td><td align="right">42 B</td></tr><tr><td>UD-IQ1_XXXS</td><td>Q1_0</td><td align="right"><strong>1.1875</strong></td><td align="right">256</td><td align="right">8</td><td align="right">38 B</td></tr></tbody></table>

We are still running benchmarks for the new data-types, but for other large models, we get **good results without any QAT / QAD**:

| Dtype        |     GiB |      PPL |      KLD |  top-p |
| ------------ | ------: | -------: | -------: | -----: |
| IQ1\_S       | 553.204 | 2.578876 | 0.564553 | 78.882 |
| UD-IQ1\_XS   | 513.583 | 2.931261 | 0.690161 | 75.726 |
| UD-IQ1\_XXS  | 473.961 | 3.540383 | 0.876007 | 71.284 |
| UD-IQ1\_XXXS | 434.340 | 4.488796 | 1.109944 | 66.257 |

### :llama: Run Qwen3.8 in llama.cpp

{% stepper %}
{% step %}
We need to use the specific IQ1\_XXXS branch [here](https://github.com/unslothai/llama.cpp/pull/61). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch iq1-narrow https://github.com/unslothai/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you just want to run the standard `IQ1_S` and other quants, then compile llama.cpp normally:

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub`). You can choose `Q1_0` for `IQ1_XXXS` or other quantized versions like `Q8_0` . If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

**Qwen3.8-27B:**

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-27B-GGUF \
    --local-dir unsloth/Qwen3.8-27B-GGUF \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q3_K_XL*" for 3-bit
```

**Qwen3.8-2.4T:**

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --local-dir unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --include "*Q1_0*" # Use "*IQ2_XXS*" for 2-bit
```

{% endstep %}

{% step %}
To run the model in llama-cli, follow the code snippets below:\
Remember to [change settings](#recommended-settings) according to your use-case.

**Qwen3.8-27B:**

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
```

{% endcode %}

**Qwen3.8-2.4T:**

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-2.4T-A95B-GGUF/UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
```

{% endcode %}
{% endstep %}

{% step %}
To run the general UD-IQ1\_S you can do:

**Qwen3.8-2.4T:**

{% code overflow="wrap" %}

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --local-dir unsloth/Qwen3.8-2.4T-A95B-GGFF \
    --include "*IQ1_S*" # Use "*IQ2_XXS*" for 2-bit
```

{% endcode %}
{% endstep %}

{% step %}
Then to run it:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-2.4T-A95B-GGUF/UD-IQ1_S/Qwen3.8-2.4T-A95B-UD-IQ1_S-00001-of-00012.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
```

{% endcode %}
{% endstep %}
{% endstepper %}

### ⚡️NVFP4

Like Qwen3.6, we’re also releasing new [dynamic NVFP4 Qwen3.8](/docs/basics/nvfp4)-27B quants that run **\~1.5× faster** than BF16 checkpoints, with **better performance** and comparable file sizes. Run Qwen3.8-27B NVFP4 **1.5x faster** on **24GB VRAM.** We also added **FP8 KV cache calibration** for 2x longer context lengths! NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see [#dgx-spark-with-nvfp4-quants](#dgx-spark-with-nvfp4-quants "mention")), B200, B300 GPUs. For older GPUs, our GGUFs work well! You can run NVFP4 quants in [vLLM](#vllm) only for now (SGLang is not supported).

* [Qwen3.8-27B-**NVFP4**](https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4) quant

<table><thead><tr><th width="90" align="right">batch</th><th width="114.5999755859375" align="right">BF16 total tok/s</th><th width="122.60003662109375" align="right">NVFP4 total tok/s</th><th width="122" align="right">speedup</th><th width="137.4000244140625" align="right">BF16 per-user</th><th align="right">NVFP4 per-user</th></tr></thead><tbody><tr><td align="right">1</td><td align="right">89.8</td><td align="right"><strong>133.7</strong></td><td align="right">1.49x</td><td align="right">89.8</td><td align="right"><strong>133.7</strong></td></tr><tr><td align="right">8</td><td align="right">649.4</td><td align="right"><strong>938.8</strong></td><td align="right">1.45x</td><td align="right">81.2</td><td align="right"><strong>117.3</strong></td></tr><tr><td align="right">32</td><td align="right">1983.0</td><td align="right"><strong>2787.0</strong></td><td align="right">1.41x</td><td align="right">62.0</td><td align="right"><strong>87.1</strong></td></tr><tr><td align="right">64</td><td align="right">3048.5</td><td align="right"><strong>4407.2</strong></td><td align="right">1.45x</td><td align="right">47.6</td><td align="right"><strong>68.9</strong></td></tr></tbody></table>

See below for previous benchmarks conducted for Qwen3.6 as well comparing to other NVFP4 implementations which use 16bit activations vs our NVFP4 activations:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F8zM7cg0Xgo2pAlop1iPW%2F01_qwen36_combined_throughput.png?alt=media&amp;token=23131a96-4c41-42d7-aa16-5bfb21bb44a3" alt="" width="563"><figcaption></figcaption></figure>

All benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s.&#x20;

For accuracy benchmarks we ran KLD and Top-1 agreement over Code, Chat and many domains. NVFP4 is consistency 92% to 97% accuracy recovery vs BF16

| corpus            |      KLD mean | top-1 agreement |
| ----------------- | ------------: | --------------: |
| zh                |       0.01628 |          93.55% |
| code              |       0.02600 |          96.68% |
| refgen            |       0.03993 |          94.46% |
| chat              |       0.05818 |          92.15% |
| ja / ko / ru / es | 0.0124-0.0155 |          94-95% |

For accuracy benchmarks For Qwen 3.6, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFhOUUUlRBj3dafGnmvhK%2F03_qwen36_combined_accuracy(7).png?alt=media&amp;token=9178447f-d5ae-4489-9bdc-73b2b7b1631b" alt=""><figcaption></figcaption></figure>

For more information, you can read our [Dynamic NVFP4 quants blog](/docs/basics/nvfp4).

To run NVFP4 quants, see below for commands to run Qwen3.8-27B in [vLLM](/docs/basics/inference-and-deployment/vllm-guide) or [SGLang](/docs/basics/inference-and-deployment/sglang-guide):

#### **vLLM:**

To install vLLM in a separate venv:

{% code overflow="wrap" expandable="true" %}

```bash
uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto
```

{% endcode %}

Then to serve the 27B variant:

```shell
vllm serve unsloth/Qwen3.8-27B-NVFP4
```

To enable MTP / speculative decoding (faster decode but somewhat less throughput), use:

```bash
vllm serve unsloth/Qwen3.8-27B-NVFP4
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
```

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

{% code overflow="wrap" expandable="true" %}

```bash
sudo apt-get update
sudo apt-get install -y ffmpeg
```

{% endcode %}

#### **SGLang:**

If using SGLang, you must use **SGLang version v0.5.19** otherwise it will not work since we quantize the lm\_head to FP8.

{% hint style="info" %}
vLLM has a `CompressedTensorsW8A8Fp8` kernel which supports this, whilst SGLang v0.5.19 cannot load the FP8 lm\_head. v0.5.19 works now!
{% endhint %}

For sglang after installing the latest:

{% code overflow="wrap" %}

```bash
uv pip install sglang sglang-kernel \
  --extra-index-url https://sgl-project.github.io/whl/cu130/ \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

{% endcode %}

You can then serve it:

```bash
sglang serve unsloth/Qwen3.8-27B-NVFP4
```

To enable MTP, use:

{% code overflow="wrap" %}

```bash
sglang serve unsloth/Qwen3.8-27B-NVFP4 --speculative-algorithm EAGLE \
    --speculative-num-steps 3 --speculative-eagle-topk 1  --speculative-num-draft-tokens 4
```

{% endcode %}

### :exploding\_head:Quantization Analysis

NVFP4 quants are 1.5x faster than BF16 and retains 92 to 97% top-1 accuracy.

We used [Dynamic 3.0 GGUFs](/docs/basics/dynamic-3.0-ggufs) to make Qwen3.8-27B much better!

Top-1 accuracy plot as shown in UD-3 fro Qwen3.8-27B:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVg5FHPxOpmaJ1r3ciw4u%2Fimage.png?alt=media&amp;token=107ef044-0181-489b-b6ae-da218e16c98b" alt=""><figcaption></figcaption></figure>

And mean KLD for Qwen3.8:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Ft8BnEqBLUQGUgu2cAKKI%2Fimage.png?alt=media&amp;token=912d5b6a-540f-4ee0-9208-f5ad45776341" alt=""><figcaption></figcaption></figure>

### 📊 Benchmarks

For GGUF quantization benchmarks you can see above for our [quantization analysis](#quantization-analysis) or [Dynamic V3.0 article](/docs/basics/dynamic-3.0-ggufs).

#### Qwen3.8-**27B**

See further below for table benchmarks:

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FWijY5wYmLx70Ao2ol7Tj%2Fqwen%20benchmark%201.jpeg?alt=media&amp;token=f69ebf50-cf35-426f-a610-8eeb32151d90" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FELa2vq8Wh3oelQi6Bf07%2Fqwen%20benchmark%202.jpeg?alt=media&amp;token=2cc6d085-630d-4fa1-9b50-784b22a1fb04" alt=""><figcaption></figcaption></figure></div>

#### Text Performance

| Benchmark                                            | Qwen3.8-27B                  | Qwen3.6-27B          | Qwen3.7-Plus         | Muse Glimmer-30B | Opus4.6 Max |
| ---------------------------------------------------- | ---------------------------- | -------------------- | -------------------- | ---------------- | ----------- |
| **Coding**                                           |                              |                      |                      |                  |             |
| Agentic terminal codingTerminal Bench 2.1 (Terminus) | 73.0                         | 63.4                 | 64.0                 | 51.7             | **78.2**    |
| Agentic codingSWE-bench Pro                          | **61.7**                     | 53.5                 | 57.6                 | 51.2             | 53.4        |
| Repo-level code generationNL2Repo-Bench              | 42.3                         | 36.2                 | 41.1                 | --               | **47.6**    |
| Agentic codingDeepSWE 1.1                            | **42.2**                     | 13.3                 | 14.2                 | --               | --          |
| Software engineeringQwenSWEBench                     | **79.0**                     | 49.3                 | 59.2                 | --               | 63.8        |
| **Agent**                                            |                              |                      |                      |                  |             |
| Long-horizon office workCoWorkBench                  | **70.7**                     | 61.0                 | 65.1                 | --               | 68.2        |
| Professional job tasksJobBench                       | **33.4**                     | 21.8                 | 27.6                 | --               | --          |
| Frontier agentic tasksAgents' Last Exam              | Pass\@1**20.4**Score**42.9** | Pass\@110.6Score27.3 | Pass\@113.2Score33.6 | --               | --          |
| General                                              |                              |                      |                      |                  |             |
| Instruction followingIFBench                         | **79.5**                     | 69.1                 | 79.1                 | 77.0             | 62.5        |
| Scientific reasoningGPQA Diamond                     | 89.2                         | 87.8                 | 90.3                 | 83.5             | **91.3**    |
| Multidisciplinary reasoningHLE                       | 30.8                         | 24.0                 | 34.7                 | 22.0             | **40.0**    |
| Competitive codingLiveCodeBench v6                   | **90.3**                     | 83.9                 | 89.6                 | --               | 88.8        |

#### Qwen3.8-**2.4T-A95B**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FWP0nBgaQcnnAGjqG1jyQ%2Fqwen3.8%20bench.jpg?alt=media&amp;token=92a0d19c-51cb-4df2-a4f5-334fe08e6a21" alt=""><figcaption></figcaption></figure>


# Qwen3.8 Fine-tuning Guide

Learn how to fine-tune Qwen3.8-27B with Unsloth.

Qwen3.8-27B can now be fine-tuned and trained with reinforcement learning (RL) via [Unsloth](https://github.com/unslothai/unsloth). It is a dense 27B unified vision-language model with native text, image and video support, thinking controls and a 262K context window.

* Unsloth trains Qwen3.8 **\~1.5x faster** with **\~50% less VRAM** than FA2 setups (no accuracy loss)
* Fine-tune Qwen3.8-27B via our **free** **Kaggle notebooks**:

| [**Conversational**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Qwen3.8_\(27B\)-Conversational.ipynb\&accelerator=nvidiaTeslaT4) (you can enable Vision) | [**RL GRPO**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Muse_Glimmer_\(30B\)-GRPO.ipynb\&accelerator=nvidiaTeslaT4) |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

* **QLoRA works with 24GB** and LoRA needs >36GB
* You can also train Qwen3.8 with [reinforcement learning](#reinforcement-learning-rl) (RL) free or on 24GB VRAM.
* **Full fine-tuning (FFT)** works as well, but it will use 4x more VRAM.
* Unsloth utilizes **Flash Linear Attention kernels** for efficient Qwen3.8 training
* Use [`Qwen3.8-27B-unsloth-bnb-4bit`](https://huggingface.co/unsloth/Qwen3.8-27B-unsloth-bnb-4bit) for 4-bit QLoRA, then **export** to NVFP4, FP8, GGUF etc.
* If you want to preserve reasoning ability, mix reasoning-style examples with direct answers and keep at least 75% reasoning data.

Please use the latest Transformers v5. Qwen3.8 uses the `qwen3_5` architecture. The first run can take longer while the Gated DeltaNet kernels compile.

{% hint style="success" %}
For free Qwen3.8-27B fine-tuning via [Unsloth](https://github.com/unslothai/unsloth), we provide multiple Kaggle notebooks that offer **30 hours of free GPU usage with 2× Tesla T4 GPUs**. Kaggle is a Google product, similar to Google Colab, and provides a convenient way to run fine-tuning workflows without needing your own GPU hardware.
{% endhint %}

### 🦥 Unsloth Guide

{% columns %}
{% column %}
Qwen3.8 can be run and fine-tuned in [Unsloth](/docs/new/studio) Desktop, our new open-source web UI for local AI.

With Unsloth Studio, you can run models locally on **MacOS, Windows**, Linux and train NVIDIA GPUs. Intel, MLX and AMD training support coming this month.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FD7qpbGJLY5YMv0E0QgrB%2Ftrainingrun_LLM-unsloth.gif?alt=media&amp;token=feb766a7-1a7b-4f47-b5f4-b38e3a0d1470" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Train Qwen3.8

Go to the Train tab then search for Qwen3.8-27B in the search bar and select your desired model and dataset. Next, adjust your hyperparameters, context length as desired.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FD7qpbGJLY5YMv0E0QgrB%2Ftrainingrun_LLM-unsloth.gif?alt=media&amp;token=feb766a7-1a7b-4f47-b5f4-b38e3a0d1470" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Monitor training progress

After you click start training, you will be able to monitor and observe the training progress of the model. The training loss should be steadily decreasing.\
Once done, the model will be automatically saved.
{% endstep %}

{% step %}

#### Export your fine-tuned model

Once done, Unsloth Studio allows you to export the model to GGUF, safetensor etc formats.
{% endstep %}

{% step %}

#### Compare fine-tuned model vs original model

Click on `Compare Mode` to compare the LoRA adapter and the original model.
{% endstep %}
{% endstepper %}

### SFT recipe

Below is a minimal SFT recipe for text-only fine-tuning. Your dataset needs a `text` column already rendered with the Qwen chat template.

```python
from unsloth import FastModel
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

max_seq_length = 2048

dataset = load_dataset(
    "json",
    data_files = "train.jsonl",
    split = "train",
)

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3.8-27B-unsloth-bnb-4bit",
    max_seq_length = max_seq_length,
    load_in_4bit = True,
    full_finetuning = False,
    offload_embedding = True,
)

model = FastModel.get_peft_model(
    model,
    finetune_vision_layers     = False,
    finetune_language_layers   = True,
    finetune_attention_modules = True,
    finetune_mlp_modules       = True,

    r = 16,
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    use_rslora = False,
    loftq_config = None,
)

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    args = SFTConfig(
        dataset_text_field = "text",
        max_seq_length = max_seq_length,
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 100,
        learning_rate = 2e-4,
        logging_steps = 1,
        optim = "adamw_8bit",
        output_dir = "outputs_qwen38",
        seed = 3407,
        dataset_num_proc = 1,
        report_to = "none",
    ),
)

trainer.train()
```

`offload_embedding=True` is optional and reduces resident VRAM by keeping the large, untied input embedding in RAM. Unsloth disables it automatically on unsupported platforms.

If you OOM, reduce `max_seq_length` and keep batch size 1 with `"unsloth"` gradient checkpointing.

### Vision fine-tuning

Qwen3.8 supports native image and video inputs. For multimodal training, enable the vision layers and use a conversational vision dataset with `UnslothVisionDataCollator`.

```python
model = FastModel.get_peft_model(
    model,
    finetune_vision_layers     = True,
    finetune_language_layers   = True,
    finetune_attention_modules = True,
    finetune_mlp_modules       = True,

    r = 16,
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    use_rslora = False,
    loftq_config = None,
)
```

Set `finetune_vision_layers=False` when your dataset is text-only.

### Reinforcement Learning (RL)

Qwen3.8 uses the same `qwen3_5` architecture, so use the Qwen3.5 Unsloth RL path and disable fast vLLM inference:

```python
from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3.8-27B-unsloth-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
    fast_inference = False,
)
```

### Saving / export fine-tuned model

Use the same chat template and EOS token during deployment that you used during training.

**Save to GGUF**

```python
model.save_pretrained_gguf(
    "qwen38_gguf",
    tokenizer,
    quantization_method = "q4_k_m",
)

# model.push_to_hub_gguf(
#     "hf_username/qwen38_gguf",
#     tokenizer,
#     quantization_method = "q4_k_m",
# )
```

**Save to vLLM**

```python
model.save_pretrained_merged(
    "qwen38_finetuned",
    tokenizer,
    save_method = "merged_16bit",
)
```

To save only the LoRA adapters:

```python
model.save_pretrained("qwen38_lora")
tokenizer.save_pretrained("qwen38_lora")
```


# Qwen3.8-Flash-Next: How to Run Locally

Guide to run Qwen3.8-Flash-Next locally.

Qwen3.8-Flash-Next is a new open-weight, **125B parameter** MoE multimodal model from Qwen. Built on the new Qwen4 architecture, it supports a 262K context window and advanced reasoning. [Qwen3.8](/docs/models/qwen3.8)-Flash-Next outperforms Claude-4.6-Opus (Max) and can run locally on devices with **75GB RAM**/unified memory with no GPU VRAM required. To run the model, use our [GGUFs](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) via llama.cpp or [Unsloth Desktop](/docs/desktop). Thank you Qwen for day zero access.

{% columns %}
{% column %}
**1-bit is 75GB** and uses 4-bit for the Ngram / PLE. This is **79% smaller** than BF16 (355GB), and retains a **top-1 accuracy of 80%**.

<a href="/pages/Ef8lrnBt1lmETOqIFCHj#run-qwen3.8-flash-next-in-unsloth" class="button primary">Run Qwen3.8-Flash</a><a href="https://unsloth.ai/download" class="button secondary">Download Unsloth</a>

{% hint style="success" %}
[**MTP**](#mtp-guide) is here! Run Qwen3.8-Flash 1.3-1.7x Faster in [Unsloth Desktop](#run-qwen3.8-flash-next-in-unsloth)!
{% endhint %}
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1YneA51oWzaIiwep9H5I%2F1000024423.gif?alt=media&amp;token=40a169cc-12f1-403a-898c-b310f81ff52a" alt=""><figcaption><p>4-bit Qwen3.8-Flash running in Unsloth</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### :gear: Usage Guide

Whether you run **Qwen3.8-Flash-Next** on a CPU with system RAM or on a GPU with VRAM may make relatively little difference. Its unique architecture allows inference using RAM or unified memory to achieve performance closer to that of GPU VRAM than is typical for other models. This makes it particularly well suited to Macs, NVIDIA DGX Spark systems, and other devices with large memory capacities.

You will need at least **75 GB of RAM or unified memory** to run the model. Its smallest 1-bit quantized version is larger than usual because of new Ngram layers or per layer embeddings which is like a lookup table. However, this also means the quantization is less aggressive, allowing the model to retain more of its original accuracy than more heavily quantized models. You can also offload the PLE / Ngram layer to SSD and use mmap which allows less usage of CPU and GPU VRAM.

#### Qwen3.8-Flash-Next Requirements:

The smallest quant works on 75GB RAM so it's best to have a 96GB RAM/unified memory device.\
**Table: Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

<table><thead><tr><th>1-bit</th><th>2-bit</th><th>3-bit</th><th>4-bit</th><th width="128">5-bit</th><th>8-bit</th><th>BF16</th></tr></thead><tbody><tr><td>75 GB</td><td>79 GB</td><td>90 GB</td><td>96-114 GB</td><td>163 GB</td><td>200 GB</td><td>355 GB</td></tr></tbody></table>

{% hint style="info" %}
If you want to use [MTP](/docs/models/mtp) for faster inference, prepare to have 1-2GB extra headroom.
{% endhint %}

### Recommended Settings

Qwen3.8-Flash-Next is a **hybrid thinking** model with different default settings for thinking and non-thinking modes. Extra high is enabled by default so if you want shorter thinking traces, you can [adjust the thinking effort](#thinking--preserve-thinking):

| Parameter            | Thinking Mode | Instruct (non-thinking) Mode |
| -------------------- | ------------- | ---------------------------- |
| `temperature`        | 1.0           | 0.7                          |
| `top_p`              | 0.95          | 0.80                         |
| `top_k`              | 20            | 20                           |
| `min_p`              | 0.0           | 0.0                          |
| `presence_penalty`   | 0.0           | 1.5                          |
| `repetition_penalty` | 1.0           | 1.0                          |

* Context length = up to `262,144`
* Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`
* Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`

### 💡 Thinking + Preserve Thinking

{% columns %}
{% column %}
Qwen3.8-Flash-Next has **Preserve Thinking** which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. [Unsloth](#run-qwen3.8-in-unsloth-desktop) has 'Think' and Preserved Thinking toggles for Qwen3.8 (see right):
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLdgmjRrb5qhpbY9PwYe8%2FScreenshot%202026-08-14%20at%2011.26.15%E2%80%AFAM.png?alt=media&amp;token=6333f5ca-196d-46ae-9efd-2e522014e6db" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

Qwen3.8-Flash-Next comes with support for `reasoning_effort`, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:

* `xhigh` (default): for complex tasks demanding thorough analysis
* `medium`: balancing accuracy and speed
* `low`: efficient reasoning optimizing for speed and cost
* none

{% hint style="warning" %}
To change[ thinking / reasoning](#how-to-enable-or-disable-reasoning-and-thinking) effort in `unsloth run` or `llama-server`, use `--chat-template-kwargs '{"reasoning_effort":"medium"}'`

If you're on **Windows** Powershell, use: `--chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"`

Change `medium` to your desired reasoning level.
{% endhint %}

### Quantization Analysis

We ran KLD for Qwen3.8-Flash quants, and show that 80% top-1 accuracy recovery is possible with 79% less disk space usage. The new architecture uses PLE / Ngrams, and these are not quantized that heavily (4-bit minimum) since they have random access pattern, and quantizing them heavily will damage the model.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FPemX6jyt4OWohwHfcjqo%2Fqwen38_flash_unsloth_top1_accuracy_new_data.png?alt=media&amp;token=3fd7713b-e9d9-43d4-bbca-96d56df43a80" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F8bGa4A4jbGsn0qrNY6Gc%2Fqwen38_flash_unsloth_kld_new_data.png?alt=media&amp;token=45ac0e9a-c9b3-4530-90f6-ed8a2328ef08" alt=""><figcaption></figcaption></figure></div>

| quant        | GB    | top-1  | mean KLD | 99.9% KLD |
| ------------ | ----- | ------ | -------- | --------- |
| UD-IQ1\_S    | 72.5  | 77.325 | 0.396070 | 7.2126    |
| UD-IQ1\_M    | 74.5  | 79.691 | 0.314739 | 6.1965    |
| UD-Q2\_K\_XL | 78.9  | 82.715 | 0.224607 | 4.9121    |
| UD-IQ3\_XXS  | 82.0  | 85.414 | 0.165120 | 4.0375    |
| UD-Q3\_K\_XL | 90.0  | 88.315 | 0.106504 | 3.0538    |
| UD-IQ4\_XS   | 93.7  | 89.554 | 0.083630 | 2.3677    |
| UD-Q4\_K\_XL | 111.3 | 92.255 | 0.046893 | 1.5468    |
| UD-Q5\_K\_XL | 158.3 | 93.680 | 0.030415 | 1.0036    |
| UD-Q6\_K\_XL | 169.2 | 94.089 | 0.027091 | 0.8416    |
| Q8\_0        | 188.2 | 94.122 | 0.026574 | 0.8118    |

## Run Qwen3.8-Flash-Next Guide

You can now run Qwen3.8-Flash-Next in Unsloth Desktop and llama.cpp. Feel free to change quantization type.

* Hugging Face: [Qwen3.8-Flash-Next-**GGUF**](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)
* ModelScope: [Qwen3.8-Flash-Next-GGUF](https://www.modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF)

<a href="/pages/CLyZKmpoJZpdaJXhLW4v#run-qwen3.8-in-unsloth-desktop" class="button primary">Run in Unsloth Desktop</a><a href="/pages/CLyZKmpoJZpdaJXhLW4v#run-qwen3.8-in-llama.cpp" class="button secondary">Run in llama.cpp</a><a href="/docs/models/qwen3.8-next#mtp-guide" class="button primary">MTP Guide</a>

{% hint style="success" %}
Qwen3.8-Flash-Next is now available to run locally in [Unsloth Desktop](#run-qwen3.8-flash-next-in-unsloth)!
{% endhint %}

### 🦥 Run Qwen3.8-Flash-Next in Unsloth

Qwen3.8-Flash-Next now is able to run in [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally), MLX and safetensor models
* [**Self-healing** tool calling](/docs/new/studio/chat#auto-healing-tool-calling) + **web search**
* [**Code execution**](/docs/desktop#code-execution) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via MLX and llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6IXaXdTVyvbrnjehlxys%2Fkimik3.gif?alt=media&amp;token=31e1213b-d7da-46e9-bc7f-3a8c402513fc" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Search and download Qwen3.8-Flash-Next

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for Qwen3.8-Flash in the search bar and download your desired model and quant.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FiEMEgtWrGc0DMZ4FLRez%2FScreenshot%202026-08-27%20at%204.21.05%E2%80%AFAM.png?alt=media&amp;token=94ad9ccf-f882-48b0-8aaa-83e90bfc2630" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run Qwen3.8-Flash-Next

MTP is automatically enabled bit you can disable it. Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).

For example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F1YneA51oWzaIiwep9H5I%2F1000024423.gif?alt=media&amp;token=40a169cc-12f1-403a-898c-b310f81ff52a" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Serve Qwen3.8-Flash-Next with Unsloth API

You can use `unsloth run` command and serve Qwen3.8 via an API using `llama-server` runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api) or [unsloth start](/docs/integrations/unsloth-start).

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
```

{% endcode %}
{% endstep %}

{% step %}

#### Unsloth is now ready

You can also do many other things with Qwen3.8-Flash-Next via Unsloth Desktop like:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRnU2breyPzalRzIHyq8U%2FScreenshot%202026-08-28%20at%2012.12.20%E2%80%AFAM.png?alt=media&amp;token=da925810-e1d3-4c06-bf27-7caad15a2330" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### :llama: Run Qwen3.8-Flash-Next in llama.cpp

{% stepper %}
{% step %}
Install the latest version of llama.cpp. You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
To run the model, you can do:

{% code overflow="wrap" %}

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-Flash-Next-GGUF \
    --local-dir unsloth/Qwen3.8-Flash-Next-GGUF \
    --include "*UD-Q4_K_XL*" # Use "*IQ2_XXS*" for 2-bit
```

{% endcode %}
{% endstep %}

{% step %}
Then to run it:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ1_S/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
```

{% endcode %}
{% endstep %}
{% endstepper %}

### MTP Guide

Qwen3.8-Flash can run with 1.3 to **1.7x faster inference** via [MTP](/docs/models/mtp) (Multi-Token Prediction) with no accuracy degradation! MTP enables Qwen3.8-Flash to reach **170 tokens/s** on 1x RTX 6000 PRO GPU compared to the 100 token baseline. MTP speeds up inference by letting a model predict multiple upcoming tokens at once instead of generating one token per step, and is especially effective on GPUs.

To run Qwen3.8-Flash with MTP, MTP is enabled by default in [Unsloth Desktop](#run-qwen3.8-flash-next-in-unsloth) or you can use our custom llama.cpp PR.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F39BmPdGOL8IlwrdQFsGh%2Fqwen38_flash_next_unsloth_ggufs_mtp_speedup_no_mtp.png?alt=media&amp;token=77c52179-821a-406b-a5ab-fd027ebe8d30" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FF9IF8IWG2DAgfLBXzhyz%2Fqwen38_flash_next_unsloth_ggufs_mtp_decode_tokens_per_s.png?alt=media&amp;token=def0f8fb-5268-4e33-9e85-0a1fed4a1156" alt=""><figcaption></figcaption></figure></div>

Gains are smaller on devices with lower memory bandwidth, such as older Macs. We created both shared MTP modules (excludes the embed\_tokens and shares it with the main model) to save disk space, RAM and VRAM usage by around 1 to 2GB.

| MTP Type | General MTP | Shared MTP | Savings |
| -------- | ----------- | ---------- | ------- |
| BF16     | 7.77 GB     | 5.23 GB    | 2.54 GB |
| Q8\_0    | 4.14 GB     | 2.79 GB    | 1.35 GB |
| Q4\_K\_M | 2.79 GB     | 1.91 GB    | 880 MB  |

The 3-bit MTP quant works on 91GB RAM so it's best to have a 96GB RAM/unified memory device.\
**Table: MTP Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

<table><thead><tr><th>1-bit</th><th>2-bit</th><th>3-bit</th><th>4-bit</th><th width="128">5-bit</th><th>8-bit</th><th>BF16</th></tr></thead><tbody><tr><td>76 GB</td><td>80 GB</td><td>91 GB</td><td>97-115 GB</td><td>164 GB</td><td>200 GB</td><td>355 GB</td></tr></tbody></table>

#### Run MTP Qwen3.8-Flash

To run Qwen3.8-Flash with MTP, all you need to do is [**install Unsloth Desktop**](#run-qwen3.8-flash-next-in-unsloth) or update to the latest verison of Unsloth then re-download the model or download the MTP file. See further below for llama.cpp instructions.

{% columns %}
{% column %}
Unsloth Desktop Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

In Unsloth Desktop, you can also change the number of draft tokens or customize MTP. Use the advanced settings in the right sidebar, and enable "Advanced settings" and you can select MTP / Ngram speculative decoding, the number of draft tokens and more:
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FqKy281quNOdn5toIr5am%2Fimage.png?alt=media&amp;token=bf0f6f00-1192-4494-977c-1bf4fa346fa0" alt="" width="305"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

#### MTP Llama.cpp Guide

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch qwen4exp/mtp https://github.com/danielhanchen/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}

Then to download the shared MTP module:

{% code overflow="wrap" %}

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-Flash-Next-GGUF \
    --local-dir unsloth/Qwen3.8-Flash-Next-GGUF \
    --include "*mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf*"
```

{% endcode %}

And to use llama-server with it:

{% code overflow="wrap" %}

```bash
llama.cpp/llama-server \
    -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
    -md unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
    --spec-type draft-mtp --spec-draft-n-max 5
```

{% endcode %}

### 📊 Benchmarks

For GGUF quantization benchmarks you can see above for our [quantization analysis](#quantization-analysis) or [Dynamic V3.0 article](/docs/basics/dynamic-3.0-ggufs).

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJFGDUmmWUvJMD0eCUbiE%2Fqwennextbe.jpg?alt=media&amp;token=0be96d30-9f51-41f3-8e3a-3366a4fdfb93" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpQBzoQNziZtrFDHCyN3t%2Fbench2max.jpg?alt=media&amp;token=3d6adbe3-6c8b-4cb4-937f-f434ebd7f106" alt=""><figcaption></figcaption></figure></div>


# Flint: Run Locally Guide

Run Recursive Flint locally with Unsloth Dynamic GGUFs, Unsloth Desktop and llama.cpp.

Recursive Flint is a **39.8B-parameter MoE (2.5B active)**, with reasoning, multi-turn tool use and **256K context**. Flint is released as Apache-2.0 and is the **strongest MoE model** under 50B parameters on agentic coding benchmarks. Run locally in [Unsloth Desktop](/docs/desktop) or llama.cpp with Unsloth’s 3-bit GGUFs on 24GB RAM. You can run at 125+ tokens/sec on an M4 Pro MacBook Pro and 150+ on a 64GB NVIDIA RTX Spark.

Our **3-bit UD-Q3\_K\_XL GGUF** is 18.50GB and fits **24GB RAM setups** at short contexts. The full BF16 GGUF is 79.58GB, while UD-Q4\_K\_XL is 25.05GB. GGUFs: [unsloth/flint-GGUF](https://huggingface.co/unsloth/flint-GGUF)

#### ⚙️ Usage Guide

**Table: Hardware Requirements** (RAM + VRAM, or unified memory).

| 2-bit    | 3-bit    | 4-bit | 6-bit | 8-bit |
| -------- | -------- | ----- | ----- | ----- |
| 13–15 GB | 16–20 GB | 26 GB | 38 GB | 44 GB |

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FGXYEQqEXseTna162HHNa%2Fflint%20pic.webp?alt=media&amp;token=bcddff05-8e5d-4026-a05e-02cab135af2e" alt="" width="563"><figcaption><p>Flint running inside of Unsloth.</p></figcaption></figure>

#### Recommended Settings

[Recursive recommends](https://huggingface.co/RecursiveSI/flint) the following sampling settings:

| Parameter       | Setting                          |
| --------------- | -------------------------------- |
| Temperature     | 1.0                              |
| Top-p           | 0.95                             |
| Top-k           | 20                               |
| Maximum context | 262,144 tokens (prompt + output) |

**Chat template:** Use the template bundled with the GGUF. For custom multi-turn clients, preserve returned `reasoning_content` and tool-call messages in the conversation history, as described in the model card.

### Run Flint Guide

We'll use **UD-Q4\_K\_XL (25.05GB)**. If memory is limited, select **UD-Q3\_K\_XL (18.5GB)** instead.

You can also read Recursive's official [guide for Flint here](#usage-guide).

#### 🦥 Run Flint in Unsloth Desktop

Run Flint in [Unsloth Desktop](https://unsloth.ai/docs/desktop) on **macOS, Windows and Linux**.

{% stepper %}
{% step %}

#### Install Unsloth

Download and launch the latest [Unsloth Desktop app](https://unsloth.ai/download) for your device:

* [Download for macOS](https://unsloth.ai/download/mac)
* [Download for Windows](https://unsloth.ai/download/windows)
* [Download for Linux](https://unsloth.ai/download/linux)

For manual installation on macOS, Linux or WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

For Windows PowerShell:

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Download and run Flint

1. Open **Model hub** or the **Select model** dropdown in Chat.
2. Search for `unsloth/flint-GGUF`, select the Unsloth repository, and download `UD-Q4_K_XL` or a smaller quant.

Flint accepts **text input**. See our [Unsloth Chat guide](https://unsloth.ai/docs/new/studio/chat) for model loading and inference settings.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FGXYEQqEXseTna162HHNa%2Fflint%20pic.webp?alt=media&amp;token=bcddff05-8e5d-4026-a05e-02cab135af2e" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Connect a coding agent

After installing Unsloth and your preferred agent, run this from your project folder:

```bash
unsloth start claude \
    --model unsloth/flint-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
```

This uses a larger context for coding tasks and requires more memory than the 8K example. You can replace `claude` with another supported agent, such as `opencode`. See [Unsloth Start](https://unsloth.ai/docs/integrations/unsloth-start) for setup and supported integrations.
{% endstep %}

{% step %}

#### Serve Flint with Unsloth API

Use `unsloth run` to load the quant and serve it locally. The `:UD-Q4_K_XL` suffix chooses the quantization:

```bash
unsloth run --model unsloth/flint-GGUF:UD-Q4_K_XL \
    --parallel 1 --alias flint \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
    --host 127.0.0.1 --port 8001
```

Use the endpoint and authentication details shown by Unsloth. For Desktop API access, create a key in **Settings → API** and follow our [API guide](https://unsloth.ai/docs/basics/api) to connect your client.
{% endstep %}
{% endstepper %}

### 📊 Quantization Analysis

The benchmarks below from the [Flint GGUF README](https://huggingface.co/unsloth/flint-GGUF) compare quantization quality against model size. **Lower KLD is better; higher top-1 accuracy is better.** These measure how closely each quant matches the reference model. Read more about [Unsloth Dynamic 3.0 GGUFs](https://unsloth.ai/docs/basics/unsloth-dynamic-v3.0-gguf).

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FR3k6FK1PyIJuU7qBx6h8%2Fsize_vs_top1(3).png?alt=media&amp;token=1ffcb4fc-696f-4fec-92a8-68b2c160246b" alt="" width="563"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FtU9zTQiXR66yrHnGX19y%2Fsize_vs_kld(3).png?alt=media&amp;token=0d18a630-93d4-448f-b00b-df41000d61de" alt="" width="563"><figcaption></figcaption></figure></div>

### 🦙 Run Flint in llama.cpp

{% stepper %}
{% step %}

#### Build llama.cpp

Use the [latest llama.cpp](https://github.com/ggml-org/llama.cpp) with Flint support. The commands below use a macOS/Linux/WSL shell. Install Git, CMake and a C++ compiler first. For NVIDIA, also install the CUDA toolkit.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch iq1-narrow https://github.com/unslothai/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

**Apple Silicon:** replace `-DGGML_CUDA=ON` with `-DGGML_CUDA=OFF`; Metal is enabled by default. For CPU-only builds, also set `-DGGML_METAL=OFF` on macOS. See the [build instructions](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) for other platforms.
{% endstep %}

{% step %}

#### Download the Unsloth GGUF

From the same working directory, download the exact 4-bit file:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -U huggingface_hub
hf download unsloth/flint-GGUF \
    flint-UD-Q4_K_XL.gguf \
    --local-dir models/flint
```

For the smaller 2-bit model, replace the filename with `flint-UD-IQ2_XXS.gguf` in both the download and run commands. If you choose BF16, download all three shards into the same folder and point the runtime at the first shard.
{% endstep %}

{% step %}

#### Chat with Flint

```bash
./llama.cpp/llama-cli \
    --model models/flint/flint-UD-Q4_K_XL.gguf \
    --ctx-size 8192 --n-gpu-layers 99 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
    --prompt "Write a Python function that merges overlapping intervals and include tests."
```

`--jinja` uses the model's chat template. Reduce `--n-gpu-layers` if VRAM is limited, or set it to `0` for CPU inference. Keep enough system RAM for layers that are offloaded.
{% endstep %}
{% endstepper %}

### Benchmarks

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fvx89npzlDGi6Mq3sWqmJ%2Fflint_benchmarks_nvfp4.png?alt=media&amp;token=759b8b48-8c15-4ca9-a1a5-3d3aab34a522" alt=""><figcaption></figcaption></figure>

| Model         | Terminal-Bench 2.1 | SWE-bench Pro v2† | SWE-bench Verified | LiveCodeBench v6 | IFBench | IFEval |
| ------------- | ------------------ | ----------------- | ------------------ | ---------------- | ------- | ------ |
| Flint BF16    | 54.9               | 89.7              | 74.2               | 80.3             | 89.2    | 95.1   |
| Flint NVFP4   | 53.7               | 88.5              | 74.1               | 80.8             | 88.1    | 94.6   |
| Qwen 3.5      | 38.7               | 68.6              | 69.2               | 74.6             | 70.2    | 91.9   |
| Qwen 3.6      | 44.9               | 73.8              | 73.4               | 80.4             | 67.9    | 89.6   |
| Laguna XS 2.1 | 24.7†              | 38.6              | 70.9               | 49.5†            | 34.4†   | 73.7†  |
| Qwen 3.8      | 73.0               | 94.9              | 75.6†              | 90.3             | 79.5    | 94.6†  |
| Muse Glimmer  | 51.7               | 86.0              | 76.0               | 63.5†            | 77.0    | 94.7†  |
| Gemma 4       | 43.4               | 61.7              | 66.6               | 80.0             | 76.0    | 94.6   |

Notes from the chart:

* † = Measured internally (as marked in the original chart).
* Flint BF16: 40B total parameters, approximately 2B active.
* Flint NVFP4: Quantized Flint checkpoint.
* Qwen 3.5, Qwen 3.6, Laguna XS 2.1: Sparse MoE category (33–35B total, 3B active).
* Qwen 3.8, Muse Glimmer, Gemma 4: Dense category (27–31B).
* All benchmark panels use a zero-based scale.


# NVIDIA Nemotron 3.5 Bolt: Run Locally Guide

Run Nemotron 3.5 Bolt locally with Unsloth GGUFs.

NVIDIA Nemotron-3.5-Bolt-**E4B** and Nemotron-3.5-Bolt-**74B-A9B** are new open models by NVIDIA. Bolt-74B-A9B is a **74B parameter**, 9B active hybrid reasoning MoE model compressed from Super VL. It supports **text, images and video**, tool calling and a **128K context** window. Run the model locally via [Unsloth Desktop](/docs/desktop) or llama.cpp.

Bolt-E4B-4GB is an **8B parameter** (4B effective), model based on [Gemma 4](/docs/models/gemma-4) E4B. It supports a **64K context window**, multi-turn tool calling and configurable reasoning. Downloads: [74B-A9B Unsloth GGUFs](https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF) · [E4B NVIDIA GGUF](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB).

<a href="/docs/models/nemotron-3.5-bolt#run-nemotron-35-bolt-guide" class="button primary">Run Nemotron-3.5-Bolt Guide</a><a href="/docs/models/nemotron-3.5-super-vl" class="button secondary">Nemotron-3.5-Super-VL Guide</a>

### ⚙️ Usage Guide

**Nemotron 3.5 Bolt Requirements**

For **74B-A9B** short-context use, budget around **32GB RAM or unified memory**; **48–64GB** gives more headroom. **E4B** runs on roughly **4GB VRAM** or **6GB RAM.**

| Model / quantization           | Download size | Memory guidance         |
| ------------------------------ | ------------- | ----------------------- |
| **E4B** · Q4\_K\_M (4-bit)     | 6.18GB        | \~4GB VRAM + system RAM |
| **74B-A9B** · IQ2\_XXS (2-bit) | 22GB          | 24–32GB RAM             |

* Memory use varies with context and runtime. E4B’s \~4GB figure is VRAM, not total system RAM or download size. The 74B-A9B model needs extra memory for images/video and the vision projector. Download sizes are for the GGUF weights.

{% hint style="info" %}
**74B-A9B** uses a **LiLiCorr** draft model in **NVFP4** (998MB) by default in Unsloth for faster output. It needs about 1–2GB more memory. E4B has a separate optional [DFlash drafter](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash).
{% endhint %}

#### Recommended Settings

Unsloth automatically configures inference settings. The 74B-A9B examples use default sampling settings. For E4B, NVIDIA recommends **temperature 1.0** and **top\_p 0.95** with reasoning enabled or disabled.

* **Context:** Up to **128K for 74B-A9B** or **64K for E4B**.

### Run Nemotron 3.5 Bolt Guide <a href="#run-nemotron-35-bolt-guide" id="run-nemotron-35-bolt-guide"></a>

We’ll use the **22GB 74B-A9B IQ2\_XXS** quant in this tutorial.

#### 🦥 Run Nemotron 3.5 Bolt in Unsloth Desktop <a href="#run-nemotron-35-bolt-in-unsloth-desktop" id="run-nemotron-35-bolt-in-unsloth-desktop"></a>

Run Nemotron 3.5 Bolt directly in [Unsloth Desktop](https://unsloth.ai/docs/desktop) on **macOS, Windows and Linux**. Search, download and chat with the model in the app.

{% stepper %}
{% step %}

#### **Install Unsloth**

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

You can also install manually:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### **Setup and Run Nemotron 3.5 Bolt**

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for **Nemotron 3.5 Bolt** in the search bar and download your desired model and quant. Now you're ready to run the model!

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

To try image understanding, attach a PNG or JPEG and ask **“What is this image?”**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FREZvGu7WyeaPxOjmrzPQ%2Fnemotron-bolt-unsloth-desktop.png?alt=media&amp;token=79d73c34-27fa-43b0-8362-99718df43478" alt="" width="375"><figcaption></figcaption></figure>

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).
{% endstep %}

{% step %}
Serve Nemotron 3.5 Bolt with Unsloth API

Serve Nemotron 3.5 Bolt with `unsloth run`. You can pass llama-server flags for context size, GPU layers and other settings. Change model name accordingly. See our [API docs](https://unsloth.ai/docs/basics/api) or [unsloth start](https://unsloth.ai/docs/integrations/unsloth-start).

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF:IQ2_XXS \
    --ctx-size 4096 --parallel 1 --alias nemotron-bolt \
    --host 127.0.0.1 --port 8001
```

{% endcode %}
{% endstep %}

{% step %}

#### Unsloth is now ready

You can also do many other things with Nemotron 3.5 Bolt via Unsloth Desktop like:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally
  {% endstep %}
  {% endstepper %}

#### 🦙 Run Nemotron 3.5 Bolt in llama.cpp <a href="#run-nemotron-35-bolt-in-llama.cpp" id="run-nemotron-35-bolt-in-llama.cpp"></a>

{% stepper %}
{% step %}
**Build llama.cpp**

Install the latest version of llama.cpp. You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

Video needs both `ffmpeg` and `ffprobe` on your PATH. Continue the commands below from this `llama.cpp` directory.
{% endstep %}

{% step %}
**Download the Unsloth GGUF**

Download `IQ2_XXS` and the vision projector for image/video input. Change the quant name below to download another version:

{% code overflow="wrap" %}

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -U huggingface_hub

hf download unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF \
    --local-dir models/nemotron-bolt \
    --include "*IQ2_XXS*.gguf" "*mmproj*.gguf"
```

{% endcode %}

Set the paths to the downloaded model and its matching projector. For a split GGUF, keep every shard together and point to the **first shard**. Adjust the filenames below to match your download.

```bash
MODEL_FILE="models/nemotron-bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-IQ2_XXS.gguf"
MMPROJ_FILE="models/nemotron-bolt/mmproj-BF16.gguf"
```

{% endstep %}

{% step %}
**Run text and images**

For text:

```bash
./llama.cpp/llama-cli \
    --model "$MODEL_FILE" --ctx-size 4096 \
    --prompt "Explain what a vision-language model does."
```

For an image, set the path to your own PNG or JPEG:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-mtmd-cli \
    --model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
    --ctx-size 4096 \
    --image /path/to/picture.jpg \
    --prompt "Describe this image and extract any visible text."
```

{% endcode %}
{% endstep %}

{% step %}
**Run video**

Start the multimodal CLI:

```bash
./llama.cpp/llama-mtmd-cli \
    --model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
    --ctx-size 4096
```

Then enter these lines **inside the CLI**:

```
/video /path/to/video.mp4
Summarize the video and describe the key events.
```

Supports MP4 clips up to 2 minutes. Use up to 1 FPS / 128 frames at 1080p, or 2 FPS / 256 frames at 720p. Start with a short clip and increase context as needed.
{% endstep %}
{% endstepper %}

### Benchmarks

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHf4Zgb51geMaxzCKvfaF%2Faccuracy.png?alt=media&amp;token=26ede764-8518-4867-8427-0e4171187747" alt=""><figcaption></figcaption></figure>


# GLM-5.3-Flash: How to Run Locally

Run the new GLM-5.3-Flash aka ox-alpha model by Z.ai.

GLM-5.3-Flash, also known as **`ox-alpha`**, is Z.ai’s new 320B parameter (18B active) multimodal open model which **outperforms** [GLM-5.2](/docs/models/glm-5.2). GLM-5.3-Flash is the smaller version of [GLM-5.3](/docs/models/glm-5.3) and rivals **Claude Opus 4.8** on coding and agentic benchmarks. You can now run the 1-bit model locally on 102GB RAM/VRAM or 3-bit on 128GB setups via llama.cpp or [Unsloth](https://github.com/unslothai/unsloth). Thank you Z.ai for day-zero access.

Unsloth dynamic **1-bit** (93GB) GGUFs retains **71% of top-1 accuracy** whilst being **85% smaller** vs BF16 (642GB). Dynamic 3-bit is 76% smaller and retains 87% accuracy.

​​<a href="/pages/0rzkFBbEB4pMiyscrTYW#run-glm-5.3-flash-ox-alpha-locally" class="button primary">Run GLM-5.3-Flash Guide</a><a href="https://unsloth.ai/download" class="button secondary">Download Unsloth</a>

{% hint style="success" %}
**Sep 4:** GLM-5.3-Flash now runs with [**3.3x faster inference**](#faster-inference-and-mtp-support)**!**
{% endhint %}

{% columns %}
{% column width="50%" %}
GLM-5.3-Flash was trained on 30T tokens and is built on a newly trained base model. Its hybrid sparse and linear attention architecture lowers long-context serving costs without sacrificing accuracy.

You can now run the model directly in [Unsloth Desktop](#run-glm-5.3-flash-in-unsloth).
{% endcolumn %}

{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FqPVJA6BHcCJIkQQrDvpD%2FScreenshot%202026-08-27%20at%207.38.11%E2%80%AFAM.png?alt=media&amp;token=576a6810-3f32-4e4c-a0d8-8a84cb733a52" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### :gear: Usage Guide

#### GLM-5.3-Flash Requirements:

The smallest 1-bit quant works on 100GB RAM while 3-bit works on 128GB devices like a Mac or NVIDIA DGX Spark.\
**Table: Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

| 1-bit  | 2-bit  | 3-bit      | 4-bit      | 8-bit  | BF16   |
| ------ | ------ | ---------- | ---------- | ------ | ------ |
| 100 GB | 115 GB | 128-150 GB | 162-210 GB | 350 GB | 650 GB |

### Recommended Settings

GLM-5.3-Flash has **3 thinking modes**: Low, High, and Max. Use Max Thinking for complicated tasks. In [Unsloth](#run-glm-5.2-in-unsloth-studio), you can easily select Low, High, or Max Thinking with a toggle in the chat area.

Use these settings for most use cases:

| Default Settings (Most Tasks) | DeepSWE              |
| ----------------------------- | -------------------- |
| `temperature` = 1.0           | `temperature` = 0.95 |
| `top_p` = 0.95                | `top_p` = 1.0        |

* **Maximum context window:** `1,048,576`.

#### Changing reasoning effort

GLM-5.3-Flash uses Max reasoning by default. It also supports reasoning efforts where `reasoning_effort` can be "low", "high", or "max".

### Faster Inference & MTP Support

As of Sep 4, we’ve added several improvements and optimizations to our day-zero [llama.cpp PR](https://github.com/ggml-org/llama.cpp/pull/27754). We implemented faster decoding pat plus bonus MTP support, enabling up to **3.3× faster inference** at long context lengths!

Everything works out of the box in [Unsloth Desktop](#run-glm-5.3-flash-in-unsloth), simply update to the latest version if needed. No additional modules or MTP files are required. Alternatively, you can follow our [llama.cpp](#run-glm-5.3-flash-in-llama.cpp) guide.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRKSzhmcluWnoNwUxFS40%2Fimage.png?alt=media&amp;token=872cd9b0-e968-4a0d-977c-5315d0ca3e49" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F2tUYioUQhnyCv1pJTfld%2Fimage.png?alt=media&amp;token=b8bfd0de-22c8-4f30-8bfd-36656ae3c16e" alt=""><figcaption></figcaption></figure></div>

Using GLM-5.3-Flash UD-IQ1\_S on 1xB200 and disregarding MTP first, we get:

| Test         | Baseline tok/s | Optimized tok/s |
| ------------ | -------------: | --------------: |
| pp512        |        1121.80 |          1122.0 |
| tg32         |          62.79 |           63.10 |
| tg32 @ 4096  |          53.52 |           59.50 |
| tg32 @ 16384 |          41.02 |           57.99 |
| tg32 @ 65536 |          20.66 |           48.99 |

Then once we add MTP, we see even larger payoffs especially for longer contexts. However we should stop at around n=2 as more draft tokens makes inference slower.

| prompt | MTP off |  n=2 |  n=3 |  n=5 |
| ------ | ------: | ---: | ---: | ---: |
| 4096   |    58.6 | 86.5 | 80.2 | 63.7 |
| 16K    |    55.0 |      | 77.2 |      |

For shorter context lengths, we still see speedups, albeit up to 1.6x faster.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmX36IJYj83OgfeQbCjh5%2Fimage.png?alt=media&amp;token=56e0ea43-df0b-47c4-8551-849921ef3913" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FCebn6JiIDQpQMSOz8nIX%2Fimage.png?alt=media&amp;token=1f76ea17-4970-45ae-b92e-3abce29628d7" alt=""><figcaption></figcaption></figure></div>

### 📈 Quantization Analysis

We quantized GLM-5.3-Flash down to UD-IQ1\_S 1bit (93.09GB) and it retains 71% of top-1% accuracy whilst being 85% smaller vs BF16 (641.64GB)

Dynamic 2-bit UD-Q2\_K\_XL is 109GB, is 83% smaller and retains 78% of accuracy.\
Dynamic 3-bit UD-IQ3\_XXS is 120GB, is 81% smaller and retains 82% of accuracy.\
Dynamic 4-bit UD-Q4\_K\_XL is 200GB, is 69% smaller and retains 93% of accuracy.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHB3bFxI8Cm3IOw3CRzvR%2Fglm53_flash_dynamic_ggufs_top1_accuracy_new_data.png?alt=media&amp;token=49882cca-1643-4e81-ad64-9d53c75ab93a" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FthGU2pMvyHLSsnOal90W%2Fglm53_flash_dynamic_ggufs_kld_benchmarks_new_data.png?alt=media&amp;token=204f70ee-cae2-4a63-b958-d4ec515f4ce3" alt=""><figcaption></figcaption></figure></div>

| Quant        | size   | top-1 acc | mean KLD | KLD 99.9% |
| ------------ | ------ | --------- | -------- | --------- |
| UD-IQ1\_S    | 93.09  | 70.89%    | 0.669714 | 9.1658    |
| UD-IQ1\_M    | 97.58  | 73.06%    | 0.572413 | 8.5069    |
| UD-IQ2\_XXS  | 101.84 | 76.30%    | 0.450148 | 7.5764    |
| UD-Q2\_K\_XL | 108.72 | 78.34%    | 0.380134 | 6.8412    |
| UD-IQ3\_XXS  | 120.37 | 81.63%    | 0.283772 | 5.9611    |
| UD-Q3\_K\_XL | 147.54 | 86.25%    | 0.159697 | 4.0281    |
| UD-IQ4\_XS   | 156.82 | 88.18%    | 0.116652 | 3.1014    |
| UD-Q4\_K\_XL | 199.71 | 92.22%    | 0.049294 | 1.4894    |
| UD-Q5\_K\_XL | 240.31 | 94.35%    | 0.027052 | 0.8696    |
| UD-Q6\_K\_XL | 291.83 | 95.23%    | 0.019007 | 0.6267    |

## Run GLM-5.3-Flash (Ox-Alpha) Locally

You can now run GLM-5.3-Flash (Ox-Alpha) in Unsloth Desktop and llama.cpp with our [specific PR](https://github.com/ggml-org/llama.cpp/pull/27754). We are using 3-bit `UD-IQ3_XXS` in our demos as it fits on 128GB devices. Feel free to change quantization type.

* Hugging Face: [GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF)

<a href="/pages/0rzkFBbEB4pMiyscrTYW#run-glm-5.3-flash-in-unsloth" class="button primary">Run in Unsloth Desktop</a><a href="/pages/0rzkFBbEB4pMiyscrTYW#run-glm-5.3-flash-in-llama.cpp" class="button secondary">Run in llama.cpp</a>

### 🦥 Run GLM-5.3-Flash in Unsloth

GLM-5.3-Flash can now run in [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally), MLX and safetensor models
* [**Self-healing** tool calling](/docs/new/studio/chat#auto-healing-tool-calling) + **web search**
* [**Code execution**](/docs/desktop#code-execution) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via MLX and llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6IXaXdTVyvbrnjehlxys%2Fkimik3.gif?alt=media&amp;token=31e1213b-d7da-46e9-bc7f-3a8c402513fc" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Search and download GLM-5.3-Flash

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for GLM-5.3-Flash in the search bar and download your desired model and quant.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FvOQkVli16S26FygKi2gw%2FScreenshot%202026-08-27%20at%204.18.11%E2%80%AFAM.png?alt=media&amp;token=df7914de-597e-43b9-8147-d568ccca0a51" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run GLM-5.3-Flash

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat). 1-bit running below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwmOvZRaD61XCmMDBlHOD%2FScreenshot%202026-08-27%20at%207.08.52%E2%80%AFAM.png?alt=media&amp;token=4980f172-4323-43a0-9338-fe0e2ca749b6" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Serve GLM-5.3-Flash with Unsloth API

You can use `unsloth run` command and serve GLM-5.3-Flash via an API using `llama-server` runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api) or [unsloth start](/docs/integrations/unsloth-start).

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/GLM-5.3-Flash-GGUF:UD-IQ3_XXS
```

{% endcode %}
{% endstep %}

{% step %}

#### Unsloth is now ready

You can also do many other things with GLM-5.3-Flash via Unsloth Desktop like:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNYiWxq5OX7NdPoqxp2Hu%2FScreenshot%202026-08-27%20at%2011.59.01%E2%80%AFPM.png?alt=media&amp;token=d9311caa-6935-47b3-a767-5c48e92b7c25" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### :llama: Run GLM-5.3-Flash in llama.cpp

{% stepper %}
{% step %}
We need to use our specific llama.cpp PR [here](https://github.com/unslothai/llama.cpp/pull/61). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch glm5next/upstream https://github.com/unslothai/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
To run the model, you can do:

{% code overflow="wrap" %}

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/GLM-5.3-Flash-GGUF \
    --local-dir unsloth/GLM-5.3-Flash-GGUF \
    --include "*UD-IQ3_XXS*" # Use "*IQ2_XXS*" for 2-bit
```

{% endcode %}
{% endstep %}

{% step %}
Then to run it:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/GLM-5.3-Flash-GGUF/UD-IQ3_XXS/GLM-5.3-Flash-UD-IQ3_XXS-00001-of-00004.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --chat-template-kwargs '{"reasoning_effort":"max"}'
```

{% endcode %}

Replace `UD-IQ3_XXS` with your preferred quant, such as `IQ2_XXS` for 2-bit once uploaded.
{% endstep %}
{% endstepper %}

## 📊 Benchmarks

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FfxF1H1wdQe3vKBepuhdA%2Fimage.png?alt=media&amp;token=b6e590cc-fafb-43b3-8e8b-0b318200cbcb" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FUy73TYPneV9NarmiUgJZ%2Fimage.png?alt=media&amp;token=8b344785-cb9b-4c61-bf55-4fdc77be1d0a" alt=""><figcaption></figcaption></figure></div>

| Benchmark                               | GLM-5.3-Flash | GLM-5.2 | DeepSeek-V4-Vision-Exp | Opus 4.8 | GPT-5.6 Terra | Gemini 3.7 Flash |
| --------------------------------------- | ------------- | ------- | ---------------------- | -------- | ------------- | ---------------- |
| Coding                                  |               |         |                        |          |               |                  |
| Terminal Bench 2.1                      | 84.3          | 81.0    | 83.9                   | 85.0     | 87.4          | 85.8             |
| <p>DeepSWE</p><p>v1.1</p>               | 63.4          | 46.2    | 59.3                   | 58.0     | 69.6          | 65.3             |
| NL2Repo                                 | 56.3          | 48.9    | 57.7                   | 69.7     | -             | -                |
| Agentic                                 |               |         |                        |          |               |                  |
| Toolathlon Verified                     | 78.4          | 59.9    | 75.9                   | 76.2     | 74.9          | -                |
| <p>AutomationBench</p><p>v1.0.6</p>     | 48.8          | 26.2    | 38.8                   | 41.0     | 37.2          | 52.3             |
| Agents' Last Exam                       | 26.3          | 20.4    | 27.3                   | 27.0     | 28.0          | -                |
| HLE w/ Tools                            | 55.3          | 54.7    | 55.1                   | 57.9     | -             | -                |
| GDPval-AA v2                            | 1773          | 1504    | 1675                   | 1582     | 1571          | 1527             |
| Vision                                  |               |         |                        |          |               |                  |
| OfficeQA Pro                            | 62.4          | -       | 57.9                   | 48.9     | -             | -                |
| <p>CharXiv Reasoning</p><p>w/ Tools</p> | 89.4          | -       | 80.4                   | 89.9     | 88.0          | 88.7             |
| <p>Chartography</p><p>w/ Tools</p>      | 78.0          | -       | 64.3                   | 75.0     | 68.0          | 65.0             |
| BabyVision                              | 53.4          | -       | 35.1                   | 46.8     | 61.6          | 70.9             |
| MVbench                                 | 77.8          | -       | 69.4                   | 67.1     | 75.0          | 82.2             |
| MMVU                                    | 80.5          | -       | 72.7                   | 67.4     | 75.8          | 82.3             |


# Qwen-Image-2.1: How to Run Locally

Run Qwen-Image-2.1 locally with Unsloth FP8 and GGUF quants.

**Qwen-Image-2.1** and **Qwen-Image-2.1-Turbo** generate and [edit images](#image-editing). Qwen-Image-2.1 has 7B parameters and needs **11GB VRAM** with GGUFs or 24GB with INT8/FP8.

{% columns %}
{% column width="50%" %}
Qwen-Image-2.1 combines a 7B image generator with a **Qwen3-VL 8B encoder**. The model supports native 2K generation, text rendering, transparent images and editing with up to 10 reference images.

The quants use [Unsloth Dynamic](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs) which upcasts important layers to higher precision for more accuracy. Run in [Unsloth Desktop](/docs/desktop), diffusers, stable-diffusion.cpp and more.

* Qwen-Image-2.&#x31;**:** [GGUF](https://huggingface.co/unsloth/Qwen-Image-2.1-GGUF) • [FP8](https://huggingface.co/unsloth/Qwen-Image-2.1-FP8)
* Qwen-Image-2.1-**Turbo**: [GGUF](https://huggingface.co/unsloth/Qwen-Image-2.1-Turbo-GGUF) • [FP8](https://huggingface.co/unsloth/Qwen-Image-2.1-Turbo-FP8)
  {% endcolumn %}

{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F8E4zaQZmeYBhYIXkvveM%2F01-2edit-multi-image.webp?alt=media&amp;token=ad7d102c-b304-4c75-b66d-4a68cd518a37" alt=""><figcaption><p>Image editing by FP8 Qwen-Image-1.2 via Unsloth</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### 🖥️  Memory requirements

Use the table below to choose a starting configuration for your hardware. Generate one image at the listed resolution, then increase it if memory allows.

FP8 quants are recommended for GPUs with 24 GB of VRAM or more. It may also be preferable for any setup with a GPU, as it can offer faster inference than GGUF models even with offloading. Just ensure you have enough RAM for offloading.

Use GGUF models mostly if you have CPU RAM, or use a unified-memory system such as a Mac.

**Memory figures are estimates, not tested minimums.** Memory usage varies with resolution and offloading settings.

| Available hardware                          | Starting configuration                                                                     |
| ------------------------------------------- | ------------------------------------------------------------------------------------------ |
| **12–16 GB VRAM**                           | GGUF **Q4\_K\_M**, 1024×1024, batch 1                                                      |
| **24 GB VRAM**                              | <p><strong>INT8/FP8:</strong> 512 × 512<br><strong>GGUF Q4\_K\_M:</strong> 1024 × 1024</p> |
| **6GB VRAM**                                | You can run FP8 **on just 6GB of VRAM using offloading**. Inference will be <2x slower.    |
| **CPU-only: 12–16 GB RAM**                  | GGUF **Q4\_K\_M** with the Q4\_K\_XL text encoder                                          |
| **Apple Silicon: 12–16 GB+ unified memory** | GGUF with a compatible native backend                                                      |

CPU offloading can help when VRAM is limited, but requires additional system RAM and may slow generation.

### Quantization Analysis

We show that INT8 has the lowest reasonable LPIPS (lower is better) than even FP8, so we default to using INT8.

| scheme | file size | LPIPS mean | LPIPS max | SSIM mean |
| ------ | --------- | ---------: | --------: | --------: |
| int8   | 7.26 GB   |      0.064 |     0.167 |     0.936 |
| fp8    | 7.12 GB   |      0.112 |     0.329 |     0.899 |

### ⚡ Quickstart

{% stepper %}
{% step %}

#### Setup Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Setup Qwen-Image-2.1

In this release, Qwen-Image-2.1 is available in Unsloth for text-to-image generation only.

Select **Images** from the sidebar. The **Create** an image workflow opens by default.

Open Select image model in there or go to Model hub and search for your desired Qwen-Image-2.1 GGUF or FP8 quant from the selector which will download the model.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FEFCApsTzrL81lOcdd298%2Fimage.png?alt=media&amp;token=07fa0438-55f4-46ec-b055-2f4b6532df96" alt="" width="563"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHjN84g8IYe9eRSRO4wGe%2Fqwenimage2.1.png?alt=media&amp;token=73f99133-f0d0-47c8-aaa4-a0097b5ff7b4" alt="" width="563"><figcaption></figcaption></figure></div>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FuWMvSNEkp7lnr2es7v32%2FScreenshot%202026-09-22%20at%206.23.20%E2%80%AFAM.png?alt=media&amp;token=570e743c-b6b9-4fa2-b06d-b6e4c955424a" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Generate your image

Describe the image you want to generate. You can keep the recommended settings for your first image.

Click **Generate**. The first generation may take longer while the model downloads and loads. Your result will appear in the gallery, where you can view its settings or download it.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrX8YDJ91NTwNVtFpERXX%2Fimage.png?alt=media&amp;token=fee9d4b5-cca3-4054-ad50-6aa1beba5a50" alt="" width="563"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F0JAa73bhb2PAD0692XqI%2F1000007556.jpg?alt=media&amp;token=c725d011-01c2-4a56-96bb-31034ecb77cf" alt="" width="375"><figcaption></figcaption></figure></div>
{% endstep %}
{% endstepper %}

### Image Editing

You can easily also edit photos with transparent backgrounds using Qwen-Image-2.1. Remove elements, add elements, change the vibe while keeping the same foundation/structure. Just select the Edit tab inside of Unsloth and write your prompt.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNA6VP3yqmBZhR4qVE90I%2F02-localized-annotation.webp?alt=media&amp;token=a42e809a-f4ea-4779-9213-606c772e9422" alt=""><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FS0j84jGUey0dnoQgEQzm%2F03-transparent-edit.webp?alt=media&amp;token=ce337489-f698-4c25-9c3b-3fd77bc2e48d" alt=""><figcaption></figcaption></figure></div>

### ⚙️ Recommended settings

**Use the recipe for the backend actually running the model**, not just its FP8/GGUF file extension.

| Setting                  | Unsloth GPU / Diffusers                       | Native stable-diffusion.cpp                      |
| ------------------------ | --------------------------------------------- | ------------------------------------------------ |
| **Steps**                | **40** - Qwen/Diffusers baseline              | **20** - CLI default and Unsloth's recorded test |
| **Guidance / CFG**       | **1.0** - CFG disabled                        | **6.0** - upstream sd.cpp example                |
| **Negative prompt**      | Leave blank; set guidance to `1.0` explicitly | Leave blank to reproduce the example             |
| **Sampler / scheduler**  | Keep the model's pipeline scheduler           | **Euler**                                        |
| **Flow shift**           | Keep the pipeline's scheduler configuration   | **Automatic; omit `--flow-shift`**               |
| **First-run resolution** | `1024×1024`                                   | `1024×1024`                                      |
| **Batch size**           | `1`                                           | `1`                                              |
| **Seed**                 | `42`, or another fixed comparison seed        | `42`, or another fixed comparison seed           |

#### Resolution and aspect ratios

Use dimensions divisible by **32**. Qwen's [native presets](https://huggingface.co/Qwen/Qwen-Image-2.1#supported-aspect-ratios) include:

| Shape               | Smaller test size | Qwen native preset |
| ------------------- | ----------------- | ------------------ |
| **Square**          | `1024×1024`       | `2048×2048`        |
| **Landscape 4:3**   | `1152×864`        | `2400×1792`        |
| **Portrait 3:4**    | `864×1152`        | `1792×2400`        |
| **Photo 3:2**       | `1248×832`        | `2528×1696`        |
| **Portrait 2:3**    | `832×1248`        | `1696×2528`        |
| **Widescreen 16:9** | `1536×864`        | `2752×1536`        |
| **Vertical 9:16**   | `864×1536`        | `1536×2752`        |

Unsloth supports dimensions up to **2048 pixels per side**. Start at **1024 × 1024** and increase the resolution if memory allows. Lowering the step count speeds up generation but may not resolve memory limits.

#### LoRAs and ControlNet

Load compatible LoRAs to add a character, subject or visual style to your generations. Adjust the LoRA weight to control how strongly it affects the result.

Supported models can also use ControlNet to guide the structure of an image. Adjust its strength to control how closely the result follows the guidance image.

#### Gallery

Generated images are saved to the local gallery. Open a result to view its prompt and settings, select **Recipe** to restore its generation setup or download the image.

### Supported models and workflows

| What you want to do          | Models to start with              |
| ---------------------------- | --------------------------------- |
| Generate or transform images | Z-Image, Qwen-Image, FLUX.1, SDXL |
| Edit using instructions      | Qwen-Image-Edit, FLUX.1 Kontext   |
| Use reference images         | FLUX.2 klein                      |

More models are available in the model picker. Formats and workflows vary by model, and unsupported workflows are hidden automatically.

### Generation settings

The default settings are a good starting point. However, it is possible to adjust settings to get the best result, these are the main parameters you may want to change:

| Setting                         | What it changes                                                                              |
| ------------------------------- | -------------------------------------------------------------------------------------------- |
| **Prompt**                      | Describes the image you want to create. In **Edit**, this becomes the instruction to follow. |
| **Aspect ratio and resolution** | Set the shape and size of the image. Larger images use more memory and take longer.          |
| **Steps**                       | Control how long the model spends generating. More steps do not always improve the result.   |
| **Guidance**                    | Controls how strongly the prompt guides the result. Some models work best at `0`.            |
| **Seed**                        | Helps recreate a result. Leave it empty to use a random seed.                                |

<details>

<summary>Advanced generation settings</summary>

These controls change how the model runs on your device. Adjust them to reduce memory usage, improve performance or troubleshoot generation.

| Setting         | What it changes                                                                                                                                  |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Speed**       | Controls compilation and performance optimizations. Compiled modes may take longer during the first generation.                                  |
| **Precision**   | Changes the numerical format used to run the model. Lower precision can reduce memory usage. Available options depend on the model and hardware. |
| **Attention**   | Selects the attention implementation used during generation.                                                                                     |
| **Memory**      | Balances generation speed against GPU memory usage. Try **Low VRAM** if a model is close to your memory limit.                                   |
| **Step cache**  | Reuses some calculations between diffusion steps to improve generation speed.                                                                    |
| **CPU offload** | Moves parts of the model into system memory. This reduces GPU memory usage but may make generation slower.                                       |

</details>

### ⚠️ Troubleshooting

<details>

<summary>Images does not appear</summary>

Update to the latest version, then restart the app. See [Updating Unsloth](https://unsloth.ai/docs/get-started/install/updating) for instructions.

</details>

<details>

<summary>A model will not download or load</summary>

Some image models are large and take time to download. Check that you have enough free storage and a stable internet connection.

Some models also require you to accept their licence on Hugging Face and add a Hugging Face token before downloading them.

</details>

<details>

<summary>The model runs out of memory</summary>

Try a smaller GGUF quantization or a 4-bit version of the model. For GGUF models, start with the size marked **recommended**.

You can also:

* Lower the image resolution.
* Keep the batch size at `1`.
* Close other applications using the GPU.
* Choose a smaller model.

A **TIGHT** model may use system memory and run more slowly. A model marked **OOM** is unlikely to fit.

</details>

<details>

<summary>A workflow is missing</summary>

The available workflows depend on the loaded model. Unsupported workflows will not appear.

For example, instruction-based editing requires a model such as Qwen-Image-Edit or FLUX.1 Kontext.

</details>

### ❓ FAQ

<details>

<summary>Does image generation run locally?</summary>

Yes. Once a model is downloaded, image generation runs on your device.

</details>

<details>

<summary>What is the difference between Transform and Edit?</summary>

**Transform** redraws an existing image using a prompt that describes the complete result.

**Edit** follows instructions such as changing a background or adding an object. It requires a compatible editing model.

</details>

<details>

<summary>Can I train my own image LoRA?</summary>

Yes. Fine-tune a supported model on your own images, then load the finished LoRA in **Create**.

See Fine-tune an image model to learn more.

</details>


# EmbeddingGemma 2 - Run Locally

Run EmbeddingGemma 2 locally and fine-tune the model with Unsloth.

**EmbeddingGemma 2** is Google’s new embedding model for text, code, images, video and audio. The model has **740M parameters**, combining a **270M** parameter **text** model with modular **vision (170M)** and **audio (300M)** encoders. It supports an Apache 2.0 license, **100+ languages** and an **8K context window** for tasks like semantic search, RAG and classification. For text and code only, you can load the smaller **270M text model**.

You can run EmbeddingGemma 2 locally via our **GGUFs** using [**Unsloth Desktop**](#use-embeddinggemma-2-in-unsloth-desktop) or even **fine-tune the model** via Unsloth. [**embeddinggemma-2-GGUF**](https://huggingface.co/unsloth/embeddinggemma-2-GGUF)

<a href="/docs/models/embeddinggemma-2#use-embeddinggemma-2-in-unsloth-desktop" class="button primary">Run EmbeddingGemma 2 Guide</a><a href="/docs/models/embeddinggemma-2#fine-tune-embeddinggemma-2-with-unsloth" class="button secondary">Fine-tuning Guide</a>

### ⚙️ Usage Guide

| Feature              | EmbeddingGemma 2                                                                        |
| -------------------- | --------------------------------------------------------------------------------------- |
| Parameters           | <p>740M total (can choose which to use):</p><p>270M text + 170M vision + 300M audio</p> |
| Inputs               | Text, code, images, video, audio and mixed inputs                                       |
| Context length       | 8,192 tokens, shared across all inputs                                                  |
| Embedding dimensions | 768, 512, 256 or 128                                                                    |

#### Recommended Settings

* **Precision:** BF16 on supported hardware, or FP32 elsewhere.
* **Dimensions:** Start with **768**. Use **512 or 256** to reduce vector storage; evaluate **128** for text workloads.
* **Normalization:** L2-normalize vectors, including after truncating them.
* **Prompts:** Use a search prompt for queries and document formatting for the content you index.

{% hint style="warning" %}
**Do not use FP16.** EmbeddingGemma 2 can produce NaN or degraded embeddings in FP16. Use BF16 or FP32, including a compatible compute dtype when loading quantized weights.
{% endhint %}

### Run EmbeddingGemma 2 Guide

<a href="/docs/models/embeddinggemma-2#use-embeddinggemma-2-in-unsloth-desktop" class="button primary">Unsloth Desktop Tutorial</a><a href="/pages/hj6zQkX4eMxnihJuV4CD#run-embeddinggemma-2-ggufs-in-llama.cpp" class="button secondary">Llama.cpp Tutorial</a>

#### 🦥 Use EmbeddingGemma 2 in Unsloth Desktop

Use EmbeddingGemma 2 to index and search your documents locally on **macOS, Windows and Linux**.

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Search the embedding model

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FA5YLVJdX7p1jXPhQ7MMr%2Fimage.png?alt=media&amp;token=b32b20b1-3c65-4c0e-b561-93f66eb30e60" alt="" width="563"><figcaption></figcaption></figure></div>

In **Settings**, open **General** and scroll to **Documents & RAG**. Choose **EmbeddingGemma 2** from the **Embedding model** dropdown, then click **Download** if needed.

You can also use our GGUF upload if needed.
{% endstep %}

{% step %}

#### Add your documents

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fo3Mu42tY5xZ9XtitMgSq%2Fimage.png?alt=media&amp;token=823216c4-1d79-432e-b428-dacc76497fc7" alt="" width="563"><figcaption></figcaption></figure></div>

Open a chat, click **+**, and enable **Chat with Files**. Then click **Add files to chat with** and choose your documents.

Unsloth uses your selected embedding model to make the documents searchable.

{% hint style="info" %}
Changing the embedding model only affects newly indexed documents. Re-upload existing documents to use the new model.
{% endhint %}
{% endstep %}
{% endstepper %}

For more ways to run EmbeddingGemma 2 with Unsloth, follow the **Python** or **GGUF** guides below.

### 📓 Try our notebooks

Run and train EmbeddingGemma 2 in Google Colab with our step-by-step notebooks.

| Notebook                                                                                                                                          | What you can do                                               |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| [Multimodal search](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29-Multimodal_Search.ipynb) | Try search across text, images, video and audio.              |
| [Text fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29.ipynb)                    | Fine-tune text embeddings for your data.                      |
| [Image-text fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29-Image_Text.ipynb)   | Train on images and captions, then compare retrieval quality. |
| [Audio fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29-Audio.ipynb)             | Train on audio and captions, then compare retrieval quality.  |

#### 🦙 Run EmbeddingGemma 2 GGUFs in llama.cpp

Run EmbeddingGemma 2 locally with [Unsloth GGUFs](https://huggingface.co/unsloth/embeddinggemma-2-GGUF). Use a llama.cpp build containing [EmbeddingGemma 2 support](https://github.com/ggml-org/llama.cpp/pull/30054); the commands below build the latest source for text and code embeddings.

{% stepper %}
{% step %}
**Build llama.cpp and download your GGUF**

Install Git, CMake and a C++ compiler, then run these commands in a new working directory. They build the latest llama.cpp source and download Unsloth’s 176 MB UD-Q4\_K\_XL GGUF:

```bash
git clone https://github.com/ggml-org/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build --config Release -j 8 --target llama-server

pip install -U huggingface_hub

hf download unsloth/embeddinggemma-2-GGUF embeddinggemma-2-UD-Q4_K_XL.gguf \
  --local-dir embeddinggemma2-gguf
```

{% endstep %}

{% step %}
**Start the embedding server**

This example serves **text and code embeddings**, starting with a 2,048-token context:

```bash
./llama.cpp/build/bin/llama-server \
  --model "embeddinggemma2-gguf/embeddinggemma-2-UD-Q4_K_XL.gguf" \
  --alias embeddinggemma2 \
  --embeddings \
  --pooling mean \
  --ctx-size 2048 \
  --batch-size 2048 \
  --ubatch-size 2048 \
  --parallel 1 \
  --host 127.0.0.1 \
  --port 8080
```

For inputs longer than 2,048 tokens, increase `--ctx-size`, `--batch-size` and `--ubatch-size` together, up to `8192`. Each input must fit in a single physical batch.
{% endstep %}

{% step %}
**Request embeddings**

Add the task prefixes yourself when calling the API:

```bash
curl http://127.0.0.1:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "embeddinggemma2",
    "input": [
      "task: search result | query: How do I reset my password?",
      "title: none | text: Select Forgot password to receive a reset link."
    ],
    "encoding_format": "float"
  }'
```

The embeddings endpoint returns normalized vectors. Check that each vector has 768 finite values before building your index.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
Use a GGUF and runtime that preserve EmbeddingGemma 2's learned embedding projection. For images, video or audio, use a runtime with explicit support for those encoders and their processor files. The command above covers text embeddings.
{% endhint %}

### 🛠️ Fine-tune EmbeddingGemma 2 with Unsloth

Fine-tuning can improve search for your documents, code and terminology. Train with pairs of questions and relevant documents, then compare search results before and after training.

Run and train EmbeddingGemma 2 in Google Colab with our free step-by-step notebooks.

| Notebook                                                                                                                                          | What you can do                                               |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| [Multimodal search](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29-Multimodal_Search.ipynb) | Try search across text, images, video and audio.              |
| [Text fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29.ipynb)                    | Fine-tune text embeddings for your data.                      |
| [Image-text fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29-Image_Text.ipynb)   | Train on images and captions, then compare retrieval quality. |
| [Audio fine-tuning](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma2_%28300M%29-Audio.ipynb)             | Train on audio and captions, then compare retrieval quality.  |

#### Prepare the model

Save the text model from the Python example:

```python
text_model.save_pretrained("embeddinggemma2-text")
```

Start a fresh Python session and import Unsloth first. Load the saved model in 4-bit and add LoRA adapters for training:

```python
from unsloth import FastSentenceTransformer
import torch

model = FastSentenceTransformer.from_pretrained(
    model_name = "embeddinggemma2-text",
    max_seq_length = 1024,
    dtype = torch.bfloat16,
    load_in_4bit = True,
    load_in_16bit = False,
    full_finetuning = False,
)

model = FastSentenceTransformer.get_peft_model(
    model,
    r = 16,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = 32,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    task_type = "FEATURE_EXTRACTION",
)
```

#### Train and evaluate

Follow our [embedding fine-tuning guide](https://unsloth.ai/docs/basics/embedding-finetuning) to prepare your dataset and run `SentenceTransformerTrainer`.

* Apply the query and document prefixes once, as in the Python search example.
* Start with `MultipleNegativesRankingLoss` and a batch sampler that avoids duplicates.
* Set `bf16=True` and `fp16=False`.
* Test search quality on queries you kept out of training.

The guide’s **EmbeddingGemma 300M notebook** covers the earlier model. Use it as a workflow reference.

#### Save your adapters

After training, save the LoRA adapters:

```python
model.save_pretrained("embeddinggemma2-lora")
```

See the linked guide for merging adapters into a standalone model. Training on images, video or audio requires a separate workflow.

### 🚀 Run EmbeddingGemma 2 in Python

This example searches three documents and ranks them by how closely they match your question. It loads only the text model to save memory.

Install SentenceTransformers and Transformers with EmbeddingGemma 2 support:

```bash
pip install -U sentence-transformers transformers
```

The example automatically selects your device and a supported precision.

```python
import torch
from sentence_transformers import SentenceTransformer

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

text_model = SentenceTransformer(
    "unsloth/embeddinggemma-2",
    device = device,
    model_kwargs = {"torch_dtype": dtype},
    config_kwargs = {"vision_config": None, "audio_config": None},
)

query = "How do I reset my password?"
documents = [
    "Select Forgot password on the sign-in page to receive a reset link.",
    "Update your billing details from the account settings page.",
    "Enable two-factor authentication to protect your account.",
]
embedding_dim = 768

query_embedding = text_model.encode(
    [query],
    prompt_name = "SearchQuery",
    truncate_dim = embedding_dim,
    normalize_embeddings = True,
)
document_embeddings = text_model.encode(
    documents,
    prompt_name = "Document",
    truncate_dim = embedding_dim,
    normalize_embeddings = True,
)

scores = text_model.similarity(query_embedding, document_embeddings)[0]
for idx in scores.argsort(descending = True).tolist():
    print(f"{scores[idx].item():.4f} | {documents[idx]}")
```

Higher scores mean a closer match. Replace the question and documents with your own content.

#### Adjust your embeddings

**Save storage:** Set `embedding_dim` to **256** to use one-third of the embedding storage, with some loss in search accuracy. Use the same dimension for queries and documents.

**Choose a prompt:** The example uses `SearchQuery` for the question and `Document` for searchable content. For code searches, use `CodeRetrieval` for the query. To compare two texts for similarity, use `SentenceSimilarity` for both.

### 🖼️ Embed images, video and audio

Combine text and media into one searchable item. This example creates a single embedding for a product description, two photos and a video.

Run it after the setup above, replacing the filenames with your own files. Your installation also needs the model processor’s image and video dependencies.

```python
multimodal_model = SentenceTransformer(
    "unsloth/embeddinggemma-2",
    device = device,
    model_kwargs = {"torch_dtype": dtype},
    config_kwargs = {"audio_config": None},
)

listing_embedding = multimodal_model.encode(
    {
        "text": "Trail running shoes. <|image|> Mesh upper. <|image|> Grip test: <|video|>",
        "image": ["shoe.jpg", "mesh.jpg"],
        "video": "demo.mp4",
    },
    prompt = "",
    normalize_embeddings = True,
)
```

Replace the filenames with your own images and video, keeping the placeholders in the same order. The model combines them into one embedding you can search with text.

{% hint style="info" %}
Audio is disabled to save memory. To include it, remove `config_kwargs` and add an `audio` input with a `<|audio|>` placeholder. Use mono audio at 16 kHz.
{% endhint %}

Text and media share the **8K context limit**. Images, video and audio don’t need task prefixes.


# Meta Muse Spark Open - Run Locally Guide

Learn how to run the new Muse Spark Open model from Meta.

Muse Spark Open is Meta’s new **816B-parameter** MoE model (\~42.7B active), built for multimodal coding and agentic workflows. Muse-Spark-Open-**816B-A42B** supports **text, image, video, and audio inputs**, text output, and a **1M-token context window**.

This guide covers running Meta Superintelligence Labs’ Muse Spark Open with Unsloth [Dynamic GGUFs](https://sites.gitbook.com/preview/site_mXXTe/~/revisions/OKhxydnrrRuNu350geRP/basics/dynamic-3.0-ggufs) for optimal performance via [Unsloth Desktop](https://sites.gitbook.com/preview/site_mXXTe/~/revisions/OKhxydnrrRuNu350geRP/models/muse-spark-1.2#unsloth-guide) or llama.cpp. The 2-bit quant fits on devices with 256GB of RAM, VRAM or unified memory. Thanks Meta for Unsloth day-zero access. [**Muse-Spark-Open-816B-A42B-GGUF**](https://huggingface.co/unsloth/Muse-Spark-Open-816B-A42B-GGUF)

### Usage Guide

Muse Spark Open is a natively multimodal reasoning model with tool-use and test-time reasoning. Muse Spark Open was additionally trained on long-horizon coding tasks including whole-repository generation, large end-to-end projects and auto-research.

The **2-bit** `UD-IQ2_XXS` quant will fit on 256GB RAM and 8-bit requires 900GB RAM.

#### Hardware requirements

Muse Spark Open uses approximately **1.63 TB for full precision BF16 language-model weights** before runtime overhead. Unsloth Dynamic quantization supports MoE architectures and uses model-specific quantization schemes to preserve accuracy while reducing memory requirements.

**Table: Muse Spark Open hardware requirements** (units = total memory: RAM + VRAM, or unified memory).

| 1-bit      | 2-bit      | 3-bit      | 4-bit  | 6-bit  | 8-bit      |
| ---------- | ---------- | ---------- | ------ | ------ | ---------- |
| 180-212 GB | 250-316 GB | 346-410 GB | 512 GB | 640 GB | 870-950 GB |

{% hint style="info" %}
As a rule of thumb, your total available memory should at least exceed the size of the quantized model you download. llama.cpp can use partial RAM / disk offload when it does not, but generation will be slower. Higher context lengths also require additional memory.
{% endhint %}

#### Recommended Settings <a href="#thinking-settings" id="thinking-settings"></a>

Muse Spark Open is a reasoning model and supports low, medium, high and xhigh reasoning efforts.

Recommended inference parameters for Muse Spark Open:

* `temperature = 1.0`
* `top_p = 1.0`
* `top_k = 0 or disabled`

**Maximum context length**: `1,048,576`

## Run Muse Spark Open Tutorials

Because quantized **Muse-Spark-Open-816B-A42B** comes in several sizes, the recommended starting point for the models is [**Dynamic**](/docs/basics/dynamic-3.0-ggufs) &#x32;**-bit**. Muse-Spark-Open-816B-A42B[ GGUFs](https://huggingface.co/unsloth/Muse-Spark-Open-816B-A42B-GGUF).

<a href="/docs/models/muse-glimmer#unsloth-guide" class="button primary">🦥 Unsloth Guide</a><a href="/pages/1hXO5DDKxmP1zKUUnBnA#llama.cpp-guide" class="button primary">🦙 Llama.cpp Guide</a>

### 🦥 Unsloth Guide

Muse Spark Open can now be run and fine-tuned in Unsloth Desktop, our new open-source desktop app for local AI. Unsloth lets you run models locally on **MacOS, Windows**, Linux and:

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio#run-models-locally) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM

{% stepper %}
{% step %}

#### **Setup Unsloth**

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

You can also install manually:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Launch Unsloth and Search and download Muse Spark Open

On first launch go to the Model hub tab and search for Muse Spark Open in the search bar and download your desired model and quant.
{% endstep %}

{% step %}

#### Run Muse Spark Open

Inference parameters should be auto-set when using Unsloth Desktop, however you can still change it manually. You can also edit the context length, chat template and other settings. You can run GGUFs, safetensors and MLX files.

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F2pHPTTp37UsjRmbCAzfM%2Fimage.png?alt=media&amp;token=f0840bf7-e08b-4b73-bdec-24caae22a29d" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Serve Muse Spark Open with Unsloth API

You can use `unsloth run` command and serve Muse Spark Open via an API using `llama-server` runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api) or [unsloth start](/docs/integrations/unsloth-start).

```bash
unsloth run --model unsloth/Muse-Spark-Open-816B-A42B-GGUF:UD-IQ2_XXS
```

{% endstep %}

{% step %}

#### Unsloth is now ready

You can also do many other things with Muse Spark Open via Unsloth Desktop like:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally
  {% endstep %}
  {% endstepper %}

### KLD Benchmarks

We ran KLD benchmarks for Muse Spark Open GGUFs for general tasks and also chat tasks. Our UD-3 quants for Muse show UD-Q4\_K\_XL as having very small KLD (the lower the better). Q3\_K\_XL also is quite reasonable.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwQAj1r2xtQU78NJLxAce%2Fmuse_spark_open_816b_kld_vs_disk.png?alt=media&amp;token=2a97b134-0deb-414b-b855-56d25a7fd1d6" alt=""><figcaption></figcaption></figure>

For top-1% accurracy, we also ran it on general tasks and chat tasks - quants below UD-IQ2\_S do degrade for general tasks, so we generally recommend using UD-IQ2\_S as the minimum quant to use for agentic tasks. Chat style tasks work fine with the smallest quants.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwcFg5X5xzxrMiTzzzZIA%2Fmuse_spark_open_816b_top1_vs_disk.png?alt=media&amp;token=3125af4a-4ecd-47eb-8a00-12960c1239b4" alt=""><figcaption></figcaption></figure>

### 🦙 Llama.cpp Guide

For this guide we will be utilizing Dynamic 2-bit for Muse-Spark-Open-816B-A42B. See: [Muse-Spark-Open-816B-A42B GGUF](https://huggingface.co/unsloth/Muse-Spark-Open-816B-A42B-GGUF)

{% stepper %}
{% step %}
Install the specific version of llama.cpp. You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.\
\
Note <https://github.com/ggml-org/llama.cpp/pull/30616> needs to be merged for building from main to work - otherwise utilize our prebuilts at <https://github.com/unslothai/llama.cpp>

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
To run the model, you can do:

{% code overflow="wrap" %}

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Muse-Spark-Open-816B-A42B-GGUF \
    --local-dir unsloth/Muse-Spark-Open-816B-A42B-GGUF \
    --include "*UD-IQ2_XXS*" # Use "*IQ2_XXS*" for 2-bit
```

{% endcode %}
{% endstep %}

{% step %}
Then to run it:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Muse-Spark-Open-816B-A42B-GGUF/UD-IQ2_XXS/Muse-Spark-Open-816B-A42B-UD-IQ2_XXS-00001-of-00006.gguf \
    --temp 1.0 \
    --top-p 1.0 \
    --top-k 64
```

{% endcode %}
{% endstep %}
{% endstepper %}

### Official Benchmarks

Meta compares Muse Spark Open to other leading open-weight models of similar size and architecture. For Muse Spark Open, xhigh reasoning is enabled with, temperature = 1.0, top\_p=1.0, top\_k=disabled across benchmarks.

<table><thead><tr><th width="177.9913330078125" align="center">Benchmark</th><th>Muse Spark Open (xhigh)</th><th>Inkling (xhigh)</th><th>Nemotron 3 Ultra (think)</th><th>Beam</th><th>Mistral Large 4</th><th>Qwen3.8-2.4T-A95B (xhigh)</th><th>GLM 5.3 (max)</th><th>Kimi K3 (max)</th></tr></thead><tbody><tr><td align="center"><strong>General</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="center"><strong>Agentic</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="center"><strong>JobBench</strong></td><td><strong>61.8</strong></td><td>31.1</td><td>27.2</td><td>-</td><td>-</td><td>52.7</td><td>61.4</td><td>54.3</td></tr><tr><td align="center"><strong>MCP-Atlas</strong></td><td><strong>89.2</strong></td><td>76.0</td><td>63.1</td><td>78.7</td><td>-</td><td>84.5</td><td>84.2</td><td>82.3</td></tr><tr><td align="center"><strong>GDPVal-AA v2</strong></td><td>1401</td><td>1079</td><td>1016</td><td>-</td><td>1424</td><td>1613</td><td><strong>1653</strong></td><td>1537</td></tr><tr><td align="center"><strong>Artificial Analysis Intelligence Index</strong></td><td>39</td><td>26</td><td>23</td><td>-</td><td>38</td><td>40</td><td><strong>45</strong></td><td>44</td></tr><tr><td align="center"><strong>OSWorld Verified</strong></td><td>80.4</td><td>40.1</td><td>-</td><td><strong>-</strong></td><td>-</td><td>-</td><td>-</td><td><strong>84.8</strong></td></tr><tr><td align="center"><strong>Toolathlon Verified</strong></td><td>72.8</td><td>45.5</td><td>34.3</td><td><strong>-</strong></td><td>-</td><td>72.5</td><td>73.0</td><td><strong>76.5</strong></td></tr><tr><td align="center"><strong>Knowledge &#x26; Reasoning</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="center"><strong>HLE-Diamond (no tools)</strong></td><td><strong>26.1</strong></td><td>11.2</td><td>8.9</td><td>-</td><td>-</td><td>17.3</td><td>16.4</td><td>22.2</td></tr><tr><td align="center"><strong>AA Omniscience</strong></td><td><strong>25</strong></td><td>2</td><td>0</td><td>13</td><td>-5</td><td>4</td><td>14</td><td>20</td></tr><tr><td align="center"><strong>SciCode</strong></td><td>58.5</td><td>47.0</td><td>40.3</td><td>49.7</td><td>54.2</td><td>54.1</td><td>59.0</td><td><strong>59.5</strong></td></tr><tr><td align="center"><strong>HLE (w/ tools)</strong></td><td>61.0</td><td>46.0</td><td>37.4</td><td>51.2</td><td>-</td><td>56.2</td><td><strong>62.5</strong></td><td>56.0</td></tr><tr><td align="center"><strong>GPQA Diamond</strong></td><td>89.0</td><td>87.2</td><td>86.7</td><td>90.5</td><td>-</td><td><strong>93.5</strong></td><td>91.7</td><td><strong>93.5</strong></td></tr><tr><td align="center"><strong>Coding</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="center"><strong>DeepSWE 1.1</strong> Preferred harness</td><td>60</td><td>-</td><td>-</td><td>44</td><td>62</td><td>57</td><td>67</td><td><strong>68</strong></td></tr><tr><td align="center"><strong>SWE-Atlas-QnA</strong> Preferred harness</td><td>58</td><td>-</td><td>-</td><td>35</td><td>59</td><td>62</td><td>59</td><td><strong>66</strong></td></tr><tr><td align="center"><strong>Terminal-Bench 2.1</strong> Terminus 2 agent</td><td>76.4</td><td>55.1</td><td>53.9</td><td>80.1</td><td>-</td><td>82.0</td><td>83.9</td><td><strong>85.0</strong></td></tr><tr><td align="center"><strong>Multimodal</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="center"><strong>ZeroBench (w/ tools)</strong></td><td><strong>50.0</strong></td><td>14.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>41.0</td></tr><tr><td align="center"><strong>BabyVision (w/ tools)</strong></td><td>80.2</td><td>34.2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td><strong>85.7</strong></td></tr><tr><td align="center"><strong>SimpleVQA (w/ tools)</strong></td><td><strong>73.6</strong></td><td>52.3</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>71.9</td></tr><tr><td align="center"><strong>WorldVQA (w/ tools)</strong></td><td>49.4</td><td>18.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td><strong>51.0</strong></td></tr><tr><td align="center"><strong>IntPhys2</strong></td><td><strong>68.3</strong></td><td>55.2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>60.3</td></tr><tr><td align="center"><strong>VideoMMEv2</strong></td><td>61.5</td><td>36.9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td><strong>65.4</strong></td></tr><tr><td align="center"><strong>Safety</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="center"><strong>FORTRESS adversarial (static)</strong> lower is better</td><td>ARS: 5.8 ORS: 17.4</td><td>ARS: 25.0 ORS: 7.4</td><td>ARS: 26.4 ORS: 13.3</td><td>-</td><td>-</td><td>ARS: 21.6 ORS: 12.8</td><td>ARS: 28.2 ORS: 3.2</td><td>ARS: 26.7 ORS: 2.7</td></tr><tr><td align="center"><strong>StrongREJECT v2 (adaptive)</strong> lower is better</td><td><strong>0.5</strong></td><td>86.4</td><td>80.7</td><td>-</td><td>-</td><td>65.6</td><td>62.9</td><td>66.6</td></tr></tbody></table>


# Run and Serve Decision Models Locally with Unsloth: Laya + Jev API

Serve and run decision models locally with Unsloth.

Run and serve decision models like **Laya** or **Clef**, open Jev alternatives, on your local **Mac**, **Windows** and **Linux** device with [Unsloth](https://github.com/unslothai/unsloth). Decision models turns text inputs into structured decisions your apps can act on - from routing support tickets, filtering emails to automated tags and classifications.

You can use the Unsloth GUI interface to expose Laya and Clef through a TypeSafe-compatible Jev API, so existing Jev integrations can work by pointing them at your local server.

All data stays on your device and **CPU**, **unified memory**, and **GPU** setups are all supported.

<div align="center"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FDvcVZXYF1Cepw6sw3UGs%2Flaya-demo-v2.gif?alt=media&amp;token=d550d7d8-d315-4e22-a9c2-eee2a0dbbf82" alt="Real time packing demo powered by local Laya through Unsloth’s Decision API: suitcase items update as you type and change your travel plans" width="563"><figcaption></figcaption></figure></div>

Send the decision model pieces of text, define the decisions you need (yes/no, choices, or scores), and get back calibrated probabilities.

### 🖥️  Recommended Settings

| Model                      | Minimum RAM | Size   | Best for                                                                    |
| -------------------------- | ----------- | ------ | --------------------------------------------------------------------------- |
| **Multilingual** (default) | 4GB RAM     | 678 MB | Most use-cases. 100+ languages, 1024 token context.                         |
| **English**                | 5GB RAM     | 846 MB | English only. Larger encoder, 512 token context.                            |
| **Typed decisions**        | 5GB RAM     | 846 MB | English only. Invoices, security incidents, customer service, agent traces. |

CPU is the default. Switch **Run on** GPU for faster answers. The GPU keeps the model in memory until you press **Unload** or restart Unsloth. If the GPU runs out of memory, Laya falls back to CPU and keeps answering.

### ⚡ Quickstart

{% stepper %}
{% step %}

#### **Setup Unsloth**

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Setup Laya

Go to Settings → API → **Decision API** and turn on **Serve requests**. The default Laya model (678 MB) will then start to download after confirming if you want to start the download.

Your API key is on the same page under Access tokens. Or turn on **Keyless API access** → **Chat and inference** and skip the key on localhost.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F0Mt4vrBDlZMpLIP4O4L2%2Fdecision-api-loaded.png?alt=media&amp;token=921d0ba2-c60f-4fc2-82f9-b44584454ba3" alt=""><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

#### Send a decision request

It's time to send a decision request. In this example, we classify a support ticket, detect refund requests, and score urgency.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F57Wcq0T86gBgHWMt8ISt%2FScreenshot%202026-09-28%20at%206.53.48%E2%80%AFPM.png?alt=media&amp;token=88c48aef-0454-485a-8a30-ec55b0b6ad0c" alt="" width="563"><figcaption></figcaption></figure>

```bash
curl http://localhost:8888/v1/systemone \
  -H "Authorization: Bearer sk-unsloth-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "laya",
    "state": "Hi, I was charged twice for my March invoice (#4411). Please refund the duplicate today.",
    "questions": {
      "team": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "invoices, payments, refunds",
          "technical": "bugs, outages, errors",
          "sales": "pricing, new plans",
          "other": "everything else"
        }
      },
      "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
      "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["not urgent", "soon", "today"]}
    }
  }'
```

You get back:

```json
{
  "model": "laya-multilingual",
  "answers": {
    "team": {"type": "choice", "choice": "billing", "confidence": 1.0, "probabilities": {"billing": 1.0, "technical": 0.0, "sales": 0.0, "other": 0.0}},
    "refund": {"type": "noul", "noul": 0.9794},
    "urgency": {"type": "score", "score": 1.9005, "confidence": 0.698, "legend": {"0": "not urgent", "1": "soon", "2": "today"}, "probabilities": {"0": 0.004, "1": 0.0915, "2": 0.9045}}
  },
  "usage": {"input_tokens": 172, "output_tokens": 0}
}
```

The first request loads the model, which takes 10 to 20 seconds. After that, requests take well under a second on most CPUs.

`state` can be a string or any JSON. `model` accepts `laya`, `default`, or `jev-latest` for the model you picked in settings, or `laya-multilingual`, `laya-english`, `laya-typed-decisions` to name one.
{% endstep %}
{% endstepper %}

### ❓ Question types

| Type     | Use it for         | You get back                                        |
| -------- | ------------------ | --------------------------------------------------- |
| `noul`   | Yes or no          | The probability of yes, from 0 to 1                 |
| `choice` | Picking one option | The most likely option, and a probability for each  |
| `score`  | Rating on a scale  | Where the text falls on your scale, counting from 0 |

`noul` means yes/no. It returns the probability of “yes”: `0.98` means very likely, `0.5` means equally likely either way.

`choice` and `score` also return `confidence`. Close to `1` means one answer stands out. Close to `0` means the probabilities are spread evenly.

{% columns %}
{% column %}
To act on a choice, use its value in `probabilities`, not `confidence`. For example, only file a bug report when `technical` is above `0.8`. Laya and Jev calculate confidence differently, so don’t reuse your Jev confidence thresholds.

Each request supports up to **64 questions**, with up to **255 options per choice** and **10 levels per score**.
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlsX4Rv1JIcz1JuTkFIBO%2Fsuitcase-10mb.gif?alt=media&amp;token=1314fb64-166e-48b2-97cf-cb086d40fbe9" alt=""><figcaption><p>Real time packing demo powered by local Laya through Unsloth’s Decision API: suitcase items update as you type and change your travel plans</p></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### 🔁 Migrating from Jev

Install the SDK with `pip install typesafe-sdk`, set two environment variables, and your existing code talks to Unsloth:

```bash
export TYPESAFE_BASE_URL=http://localhost:8888
export TYPESAFE_API_KEY=sk-unsloth-YOUR_KEY
```

The SDK's default model, `jev-latest`, maps to whichever model you picked in Settings.

Two things differ from Jev. `confidence` uses a different formula, so a threshold tuned on Jev won't transfer; threshold on `probabilities` instead. And options in a choice share a fixed token budget, so past about 20 described options the labels get trimmed.


# NVIDIA Nemotron 3.5 Super VL: Run Locally Guide

Run Nemotron 3.5 Super VL locally with Unsloth GGUFs.

NVIDIA Nemotron-3.5-Super-VL-120B-A12B is a **120B parameter, 12B active** hybrid reasoning MoE model with **up to 1M tokens of context**. It accepts text, images and video for document understanding, computer use and agent workflows, and generates text.

It supports JSON output and tool calling, and includes a built-in Multi-Token Prediction (MTP) head. Unsloth GGUFs range from **46.1GB (1-bit) to 132GB (8-bit)**. We also made NVFP4 quants (79GB) and added the LiLiCorr DSpark module. **Unsloth quants:** [NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF)

<a href="/docs/models/nemotron-3.5-super-vl#run-nemotron-35-super-vl-guide" class="button primary">Nemotron-3.5-Super-VL Guide</a><a href="/docs/models/nemotron-3.5-bolt" class="button secondary">Run Nemotron-3.5-Bolt Guide</a>

### ⚙️ Usage Guide

**Nemotron 3.5 Super VL Requirements**

The smallest **1-bit GGUF is 46.1GB**; plan for at least **64GB of RAM or unified memory**. The **4-bit GGUF is 86.9GB**, with around **128GB** recommended. You can also split the model across system RAM and GPU VRAM. You can also run the NVFP4 which is a direct lossless conversion from NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

| Quantization         | GGUF size | Est. minimum memory |
| -------------------- | --------- | ------------------- |
| UD-IQ1\_M (1-bit)    | 46.1GB    | 64GB                |
| UD-IQ2\_XXS (2-bit)  | 51.1GB    | 64GB                |
| UD-IQ2\_M (2-bit)    | 55.9GB    | 80GB                |
| UD-Q2\_K\_XL (2-bit) | 57.8GB    | 80GB                |
| UD-IQ3\_XXS (3-bit)  | 59.8GB    | 80GB                |
| UD-Q3\_K\_XL (3-bit) | 65.8GB    | 96GB                |
| NVFP4 (4-bit)        | 79GB      | 96GB                |
| UD-Q4\_K\_XL (4-bit) | 86.9GB    | 128GB               |
| UD-Q5\_K\_XL (5-bit) | 111GB     | 160GB               |
| UD-Q6\_K\_XL (6-bit) | 121GB     | 160GB               |
| Q8\_0 (8-bit)        | 132GB     | 192GB               |

Memory figures are **planning estimates** for a modest context, including headroom for the operating system, runtime, KV cache and vision projector. GGUF file size alone is not the RAM requirement. Longer contexts and video need more memory; all model weights must be stored even though only 12B parameters are active per token. Count unified memory once.

A **128GB unified-memory system**, such as DGX Spark, fits the suggested 4-bit memory budget. NVIDIA documents DGX Spark support through NIM; these GGUF memory estimates are not measured performance results.

### Recommended Settings

NVIDIA recommends these settings:

| Parameter     | Value  |
| ------------- | ------ |
| `temperature` | `1.0`  |
| `top_p`       | `0.95` |

Start with **16K context** and increase it as memory allows. The model supports up to **1M tokens**, but your runtime or NIM profile may have a lower limit. Prompt and output tokens share this budget. The examples serve one request at a time.

**Thinking on or off**

The model supports `enable_thinking=True` (default) for reasoning and `enable_thinking=False` for direct answers. NVIDIA NIM accepts this through chat-template kwargs. For local GGUFs, use the model’s chat template and your runtime’s supported thinking controls.

{% hint style="info" %}
Nemotron 3.5 Super VL ships with **MTP (Multi-Token Prediction)**, which is enabled by default in Unsloth for faster output. You can turn it off in the settings. See our [MTP guide](https://unsloth.ai/docs/models/mtp).
{% endhint %}

### Run Nemotron 3.5 Super VL Guide <a href="#run-nemotron-35-super-vl-guide" id="run-nemotron-35-super-vl-guide"></a>

For this tutorial, we will use **UD-Q4\_K\_XL (86.9GB)** with around **128GB of memory**. Choose a smaller quant from the table above if you have less memory.

#### 🦥 Run Nemotron 3.5 Super VL in Unsloth Desktop <a href="#run-nemotron-35-super-vl-in-unsloth-desktop" id="run-nemotron-35-super-vl-in-unsloth-desktop"></a>

Run Nemotron 3.5 Super VL directly in [Unsloth Desktop](https://unsloth.ai/docs/desktop) on **macOS, Windows and Linux**. Search, download and chat with the model in the app.

{% stepper %}
{% step %}

#### **Setup Unsloth**

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

You can also install manually:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Setup and Run Nemotron 3.5 Super VL

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for Nemotron 3.5 Super VL in the search bar and download your desired model and quant. Now you're ready to run the model!

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

To try image understanding, attach a PNG or JPEG and ask **“What is this image?”**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrTV33NAoY62hmlGHaEyw%2Fnemotron-super-vl-unsloth-desktop.png?alt=media&amp;token=e38fc0f5-bae6-442f-8c6f-4e9c54d326aa" alt="" width="563"><figcaption></figcaption></figure>

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).
{% endstep %}

{% step %}

#### Serve Nemotron 3.5 Super VL with Unsloth API

Serve Nemotron 3.5 Super VL with `unsloth run`. You can pass llama-server flags for context size, GPU layers and other settings. See our [API docs](https://unsloth.ai/docs/basics/api) or [unsloth start](https://unsloth.ai/docs/integrations/unsloth-start).

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF:UD-IQ2_XXS \
    --ctx-size 4096 --parallel 1 --alias nemotron-super-vl \
    --host 127.0.0.1 --port 8001
```

{% endcode %}
{% endstep %}

{% step %}

#### Unsloth is now ready

You can also do many other things with Nemotron 3.5 Super VL via Unsloth Desktop like:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally
  {% endstep %}
  {% endstepper %}

### KLD Benchmarks

We used UD-3 to convert Nemotron Super and did general KLD and also an unseen text KLD dataset to gauge for overfitting - generally UD-Q4\_K\_XL is the best in terms of KLD vs disk space - NVFP4 is also generally ok, and UD-Q3\_K\_XL is a also reasonable.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FI0Lb9lHn00vouMw6Lj8n%2Fnemotron35_super_vl_kld_vs_disk(2).png?alt=media&amp;token=7aea3a84-5740-45e3-b542-2982133cafdb" alt=""><figcaption></figcaption></figure>

#### 🦙 Run Nemotron 3.5 Super VL in llama.cpp <a href="#run-nemotron-35-super-vl-in-llama.cpp" id="run-nemotron-35-super-vl-in-llama.cpp"></a>

{% stepper %}
{% step %}
**Build llama.cpp**

Install the latest version of llama.cpp. You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

Video needs both `ffmpeg` and `ffprobe` on your PATH. Continue the commands below from this `llama.cpp` directory.
{% endstep %}

{% step %}
**Download the Unsloth GGUF**

Download the model and its vision projector:

{% code overflow="wrap" %}

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -U huggingface_hub

hf download unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF \
    --local-dir models/nemotron-super-vl \
    --include "*UD-Q4_K_XL*.gguf" "*mmproj*.gguf"
```

{% endcode %}

Set the paths to the downloaded model and its matching projector. For a split GGUF, keep every shard together and point to the **first shard**. Adjust the filenames below to match your download.

```bash
MODEL_FILE="models/nemotron-super-vl/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-UD-Q4_K_XL.gguf"
MMPROJ_FILE="models/nemotron-super-vl/mmproj-BF16.gguf"
```

{% endstep %}

{% step %}
**Run text and images**

For text:

```bash
./llama.cpp/llama-cli \
    --model "$MODEL_FILE" --ctx-size 16384 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
    --prompt "Explain what a vision-language model does."
```

For an image, set the path to your own RGB PNG or JPEG. For PDFs, render pages to images first:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-mtmd-cli \
    --model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
    --ctx-size 16384 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
    --image /path/to/picture.jpg \
    --prompt "Describe this image and extract any visible text."
```

{% endcode %}
{% endstep %}

{% step %}
**Run video**

Start the multimodal CLI:

```bash
./llama.cpp/llama-mtmd-cli \
    --model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
    --ctx-size 16384 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0
```

Then enter these lines **inside the CLI**:

```
/video /path/to/video.mp4
Summarize the video and describe the key events.
```

The model card recommends **MP4 clips of 1–2 minutes**. NIM samples at 2 FPS, up to 256 frames; local sampling depends on the runtime. Start with a short clip and increase context as needed, since video frames consume prompt tokens.
{% endstep %}
{% endstepper %}

#### Llama-server serving and deployment <a href="#llama-server-serving-and-deployment" id="llama-server-serving-and-deployment"></a>

{% stepper %}
{% step %}
**Start llama-server**

Create a media directory and start the server:

{% code overflow="wrap" %}

```bash
mkdir -p media
./llama.cpp/llama-server \
    --model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
    --alias nemotron-super-vl \
    --ctx-size 16384 --parallel 1 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
    --media-path "$PWD/media" \
    --host 127.0.0.1 --port 8001
```

{% endcode %}

Open **<http://127.0.0.1:8001>** on the same machine to chat with the llama.cpp server.
{% endstep %}

{% step %}
**Send an API request**

For video, copy a short clip to `media/clip.mp4`, then run this command in another terminal:

{% code overflow="wrap" %}

```bash
curl http://127.0.0.1:8001/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "nemotron-super-vl",
      "messages": [{"role": "user", "content": [
        {"type": "text", "text": "Summarize this video."},
        {"type": "input_video", "input_video": {"url": "file://clip.mp4"}}
      ]}],
      "max_tokens": 1024
    }'
```

{% endcode %}

`--media-path` allows the server to read local media from that directory. For text-only requests, use `"content": "What is 2+2?"` instead of the content array.
{% endstep %}
{% endstepper %}

#### 📊 Benchmarks

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBjoqDdZWELPnsSztDMlZ%2Faccuracy_plot(1).png?alt=media&amp;token=e4994520-d7a8-4d06-b183-fd6182c878d3" alt=""><figcaption></figcaption></figure>


# DeepSeek-V4: How to Run Locally

Run DeepSeek-V4-Pro-0813 and DeepSeek-V4-Flash-0731 locally on your own device!

DeepSeek-V4, DeepSeek-V4-Flash-Vision-Exp, V4-**Pro-0813**, and V4-**Flash-0731** are new open-weight models - the Flash variant has 284B parameters (13B active), while V4-Pro has 1.6T (49B active). **V4-Pro-0813**, released on **Aug 13**, matches Claude-4.8-Opus performance, while **V4-Flash-0731**, released on **July 31**, delivers the best performance in its size class and **outperforms V4-Pro** (Preview). Built for coding, agentic, and chat workflows with a **1M context window**, this guide shows how to run DeepSeek-V4-Flash-0731 locally using Unsloth Dynamic GGUFs and [Unsloth Desktop](/docs/desktop).

For **lossless** DeepSeek, use Q8 (`UD-Q8_K_XL`), which is only **7GB larger** than Q4 (`UD-Q4_K_XL`). The lossless 8-bit GGUF is **162 GB** and 3-bit is **103GB** which can run on a **110GB RAM** devic&#x65;**.** DeepSeek-V4-Flash-0731 scores 82.7% on Terminal Bench 2.1, 54.4% on DeepSWE, and 54.2% on NL2Repo. [DSpark](#dspark-speculative-decoding) is also enabled for GGUFs, enabling up to **2x faster decoding speed**!

{% hint style="success" %}
**Aug 31:** DeepSeek-Flash-Vision-Exp is now available!

**Aug 13:** DeepSeek-**V4-Pro-0813** was released and [quants are now](https://huggingface.co/unsloth/DeepSeek-V4-Pro-0813-GGUF) available to run.
{% endhint %}

{% hint style="success" %}
**Aug 6: DSpark is enabled for DeepSeek-V4-Flash-0731 - enabling 1.5x to 1.9x faster inference! DSpark is automatically enabled in** [**Unsloth**](#unsloth-studio-guide)**.**

We also improved the [DeepSeek-V4 chat jinja template](#deepseek-v4-chat-template-improvements), and tested over 4000 conversations to be equivalent with the official baseline.
{% endhint %}

| [DeepSeek-V4-Pro-0813-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Pro-0813-GGUF) | [DeepSeek-V4-Flash-**0731**-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF) | [DeepSeek-V4-Flash-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF) |
| ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |

<a href="/docs/models/deepseek-v4#usage-guide" class="button primary">Usage Guide</a><a href="/docs/models/deepseek-v4#run-deepseek-v4-flash-tutorials" class="button primary">Running Tutorials</a>

### 📊 Quantization Analysis

Our `UD-Q8_K_XL` quant is fully lossless. DeepSeek-V4-Flash is [quantization-aware-trained](/docs/blog/quantization-aware-training-qat): the official checkpoint stores its routed experts (96% of the model) natively in MXFP4 and everything else in FP8 or BF16. GGUF's MXFP4 is exactly that format, so we repack the experts bit-for-bit, and FP8 dequantizes into BF16 with no rounding. We checked every tensor against the official DeepSeek weights: all 1,328 are bit-identical, and it stays lossless at inference (KL-divergence \~0, 100% top-token agreement).

**Non**-Unsloth DeepSeek-V4-Flash GGUFs were converted without these paths thus deviating from the official weights. `UD-Q4_K_XL` keeps the same bit-exact experts and only quantizes the non-expert tensors (4% of the model) to Q8\_0, so it sits right next to Q8 in size and quality.

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FXjXSVsiH2ZIKlJ7iZPDx%2Fkldddd.webp?alt=media&amp;token=e1db5fca-b2b4-411e-8956-29127ef6bfe0" alt="" width="563"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHoYuIu6dNNuLrM4nmdbi%2Fpareto.png?alt=media&amp;token=1c7a44bb-13c3-4fc4-8d57-fb350ffe3c14" alt="" width="563"><figcaption></figcaption></figure></div>

Measured against the official weights, both Unsloth quants are on the quality/size frontier. UD-Q8\_K\_XL is the only lossless point. UD-Q4\_K\_XL matches other community MXFP4 formats and is more accurate than the Q4\_K-experts conversions, which are larger yet land at 0.029 KLD.

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FxW5lIQID1dIRlVajHSWm%2Fper_layer.png?alt=media&amp;token=94dd4abc-e7ef-4aa8-8d2f-de8c98a69166" alt="" width="563"><figcaption></figcaption></figure></div>

The error split by layer shows why. Keeping the native MXFP4 experts means 0% weight error at every layer. Conversions that re-quantize the experts to Q4\_K or IQ2\_XXS round almost every weight: 5% for Q4\_K, over 30% for IQ2\_XXS.

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FINDS7DqyP9TUAs4Bj9rS%2Fwhy_lossless.png?alt=media&amp;token=efdb261b-f969-42b8-89ec-bc25e8bbdaf3" alt="" width="563"><figcaption><p>Our MXFP4 has exactly zero error on all 8.4M weights, while Q4_K, a different 4-bit grid, must round each one (5.2% RMSE).</p></figcaption></figure></div>

We also found using Q8\_0 and F16 for some tensors  is not lossless, and it gets worse since QAT was applied by DeepSeek to make MXFP4 / FP8 work well, so we had to leave them in BF16 directly. So use UD-Q8\_K\_XL for a true lossless quant, and UD-Q4\_K\_XL downcasts some of the BF16 items to Q8\_0.

For full benchmark tables of [GGUF Benchmarks, see here](#gguf-benchmarks).

### :speech\_balloon: DeepSeek V4 Chat template improvements

We also improved the DeepSeek-V4 chat jinja template, and tested over 4000 conversations to be equivalent with the golden baseline (official DS4)

We added `reasoning_effort` and you can select `max, high` just like official DeepSeek-V4. We prepend the correct system prompt as per DS4, and followed gpt-oss's style.

And for tool calls, `reasoning_content` was retained for DS4, but the jinja chat template would exclude them. We added it back.

#### **Disabling Thinking, changing reasoning effort**

DeepSeek-V4 uses reasoning by default. It also supports reasoning efforts where `reasoning_effort` can be "high", "max" or disabled.

To disable thinking, use `--chat-template-kwargs '{"enable_thinking":false}'`. If you're on **Windows** Powershell, use: `--chat-template-kwargs "{\"enable_thinking\":false}"`

You can also use `--reasoning on` or `--reasoning off` in llama.cpp as well now!

For reasoning effort customization and or to disable reasoning, use the below examples:

```bash
--chat-template-kwargs '{"reasoning_effort":"max"}'
--chat-template-kwargs '{"reasoning_effort":"high"}'
--chat-template-kwargs '{"enable_thinking":false}'
```

### ⚙️ Usage Guide

DeepSeek-V4-Flash is smaller and faster than DeepSeek-V4-Pro, with **284B** parameters (13B active), and a **1M context window**. The model has 3 modes, **Non-think**, **Think** **High** and **Think** **Max**.&#x20;

It's recommended to use `UD-IQ3_XXS` which is **103GB** for best results. Because the file size does not include KV cache, context allocation, try to have at least **110GB RAM** to run the model.

The `UD-Q8_K_XL` quant is DeepSeek-V4-Flash in full original precision. It is 162GB size and it's best to have at least 169GB of available RAM/VRAM available.

**Table: Inference hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

<table><thead><tr><th>Format</th><th width="129.8004150390625">1-bit</th><th width="130.85650634765625">2-bit</th><th width="140.26702880859375">3-bit</th><th>4-bit (near Lossless)</th><th>Q8_K_XL (Lossless)</th></tr></thead><tbody><tr><td>Standard</td><td>92 GB</td><td>102 GB</td><td>110-135 GB</td><td>162 GB</td><td>169 GB</td></tr><tr><td>DSpark</td><td>102 GB</td><td>112 GB</td><td>120-145 GB</td><td>172 GB</td><td>179 GB</td></tr></tbody></table>

{% hint style="info" %}
**DSpark uses more VRAM than standard**, so plan for **\~10 GB** additional RAM/VRAM headroom.
{% endhint %}

{% hint style="success" %}
For best performance, make sure your total available memory, including VRAM and system RAM, exceeds the quantized model file size by a comfortable margin.
{% endhint %}

### Recommended Settings

DeepSeek recommends these parameters for best performance: `temperature = 1.0`, `top-p = 1.0`. For **DeepSeek-V4-Flash-0731** and agentic scenarios, `top-p = 0.95` is suggested instead and `top-p = 1.0` for other tasks.

**Think High is on by default.** If disabled, you can enable it via: `--chat-template-kwargs '{"enable_thinking":true}'` or toggle it via the UI dropdown in [Unsloth](#unsloth-studio-guide). Also see [#deepseek-v4-chat-template-improvements](#deepseek-v4-chat-template-improvements "mention")

{% columns %}
{% column width="50%" %}

| DeepSeek-V4-Flash-0731        |
| ----------------------------- |
| `temperature = 1.0`           |
| `top-p = 1.0`                 |
| `top-p = 0.95` (agentic only) |
| {% endcolumn %}               |

{% column width="50%" %}

| Old DeepSeek-V4-Flash and V4-Pro |
| -------------------------------- |
| `temperature = 1.0`              |
| `top-p = 1.0`                    |
| {% endcolumn %}                  |
| {% endcolumns %}                 |

* **Maximum context window:** `1,048,576`
* For Think Max, set context to at least **384K tokens**.

## Run DeepSeek-V4-Flash Tutorials:

For this tutorial, we will use the 3-bit quant `UD-IQ3_XXS`, as it fits on a 128GB RAM device. Replace `UD-IQ3_XXS` with `UD-Q8_K_XL` (original quality) or another quant if your machine has enough memory. You can now run DeepSeek-V4-Flash-0731 in [Unsloth Desktop](#run-in-unsloth-studio) . **DSpark is automatically enabled in** [**Unsloth**](#unsloth-studio-guide)**.**

<a href="/docs/models/deepseek-v4#unsloth-studio-guide" class="button primary">🦥 Unsloth Guide</a><a href="/pages/2f4eyCpdyRknNuEtv22n#llama.cpp-guide" class="button primary">🦙 Llama.cpp Guide</a><a href="/docs/models/deepseek-v4#dspark-speculative-decoding" class="button secondary">⚡DSpark Guide</a>

### 🦥 Unsloth Guide

DeepSeek-V4-Flash-0731 can now be run and trained in [Unsloth](/docs/new/studio), our new open-source UI for local AI. Unsloth Desktop lets you run models locally on **MacOS**, **Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio#run-models-locally) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FCwcXvmLERJSUNF6iNTiY%2Fdeepseek-v4-flash-0731-unsloth-studio.png?alt=media&amp;token=6fdc0216-d21e-42e7-bb57-8e1abf6e7f4f" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Search and download DeepSeek-V4-Flash

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for DeepSeek-V4-Flash in the search bar and download your desired model and quant.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FGEc90VxPAgTHulQe9zB0%2FScreenshot%202026-07-31%20at%208.01.35%E2%80%AFAM.png?alt=media&amp;token=2931411d-8f75-4bfe-918a-73fcafe918f9" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run DeepSeek-V4-Flash-0731

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. Because **Think High is on by default**, you can go to the right dropdown to toggle it to Non-think or Think Max. You can also edit the context length, chat template and other settings. **DSpark is automatically enabled in** [**Unsloth**](#unsloth-studio-guide)**.**

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlvJqDRKlWdAVkn3HXJZA%2F1000024247.png?alt=media&amp;token=e84fd31d-7720-40d5-aba1-ba65ac34ce97" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Serve DeepSeek-V4 with Unsloth API

You can use `unsloth run` command and serve DeepSeek via an API using `llama-server` runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our [API docs](/docs/basics/api) or [unsloth start](/docs/integrations/unsloth-start).

```bash
unsloth run --model unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ3_S
```

{% endstep %}

{% step %}

#### Unsloth is now ready

You can also do many other things with DeepSeek via Unsloth Desktop like:

* **Connect tools:** [Claude Code](/docs/basics/claude-code), [Codex](/docs/basics/codex), [web search](/docs/new/studio/chat#advanced-web-search), [MCP](/docs/basics/mcp) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning) locally
  {% endstep %}
  {% endstepper %}

### 🦙 Llama.cpp Guide

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` **on** [**GitHub here**](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
You can now use `llama.cpp` directly to load and download models, just like `ollama run`. First, select the quantization type you want like `IQ3_XXS`. Also use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Note this download process might be very slow, so it's probably best to use the manual download process in the next section.

```bash
export LLAMA_CACHE="unsloth/DeepSeek-V4-Flash-0731-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ3_S \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.01
```

{% endstep %}

{% step %}
If you want to download the model manually, we can download the model via the code below (after installing `pip install huggingface_hub`). If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

```bash
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
    --local-dir unsloth/DeepSeek-V4-Flash-0731-GGUF \
    --include "*UD-IQ3_S*" # Use "*UD-IQ4_XS*" for 4-bit
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 32768` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-IQ3_S/DeepSeek-V4-Flash-0731-UD-IQ3_S-00001-of-00004.gguf \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.01
```

{% endcode %}
{% endstep %}
{% endstepper %}

## :zap:DSpark - Speculative Decoding

DeepSeek-V4-Flash-0731 has native DSpark, which allows for up to **2x faster decoding speed**! DSpark is a new algorithm by DeepSeek that is superior to naive MTP, and was introduced in this [paper](https://arxiv.org/abs/2607.05147). DSpark enables DeepSeek-V4-Flash to reach **120 tokens/s** on a B200 GPU compared to the original 60 tokens/s baseline. **DSpark is automatically enabled in the** [**Unsloth**](#unsloth-studio-guide) **local UI.**

Llama.cpp integrated DSpark as part of [PR 25784](https://github.com/ggml-org/llama.cpp/pull/25784) with further improvements to multi GPU and more. We show using `--spec-draft-n-max 3` as a good default, allowing 1.9x faster inference speed. Larger values seem to be slower.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FRV0T1cVPpCnacNnywAsL%2Fimage.png?alt=media&amp;token=b570026f-4577-4a45-936f-98a650d644ac" alt=""><figcaption></figcaption></figure>

Download both the drafter and the GGUF - we made 2 Q8\_0 and the lossless BF16 one. Note DSpark will need \~10GB more memory usage, so 128GB machines will need IQ3\_XXS and Q8\_0

{% code overflow="wrap" %}

```bash
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
    --local-dir unsloth/DeepSeek-V4-Flash-0731-GGUF \
    --include "*dspark-DeepSeek-V4-Flash-0731-Q8_0*" \
    --include "*UD-IQ3_XXS*" # Use "*UD-IQ4_XS*" for 4-bit
```

{% endcode %}

Then load it via llama-cli or llama-server:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf \
    -md unsloth/DeepSeek-V4-Flash-0731-GGUF/dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.01 \
    --spec-type draft-dspark \
    --spec-draft-n-max 3 \
    -ngl 99 -ngld 99
```

{% endcode %}

Benchmarks from the original DSpark paper as well showing how it fares against MTP:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNXmTkimQGD9JzAq4OtK9%2Fimage.png?alt=media&amp;token=bab3816c-e3cd-4bd8-a234-6226520f45f6" alt=""><figcaption></figcaption></figure>

## 📊 Benchmarks

### GGUF Benchmarks

See below for a table comparing benchmarks for quants from Unsloth and other providers. Reference = official weights. Perplexity and KL-divergence over wikitext-2 at ctx 512 on 4x B200.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FXjXSVsiH2ZIKlJ7iZPDx%2Fkldddd.webp?alt=media&amp;token=e1db5fca-b2b4-411e-8956-29127ef6bfe0" alt="" width="563"><figcaption></figcaption></figure>

| Quant                              | Size (GB) | PPL    | Mean KLD           | RMS delta-p | Same top token | Bit-exact weights |
| ---------------------------------- | --------- | ------ | ------------------ | ----------- | -------------- | ----------------- |
| Official (reference)               | 156.4     | 4.5319 | 0                  | 0%          | 100%           | 100%              |
| **Unsloth UD-Q8\_K\_XL**           | 161.9     | 4.5319 | **\~0 (lossless)** | 0.000%      | 100.000%       | **100.000%**      |
| **Unsloth UD-Q4\_K\_XL**           | 155.1     | 4.5335 | 0.0102             | 3.40%       | 96.28%         | 97.46%            |
| bartowski MXFP4                    | 156.0     | 4.5351 | 0.0105             | 3.42%       | 96.18%         | 97.57%            |
| antirez Q4KExperts-F16 (imatrix)   | 164.6     | 4.5743 | 0.0291             | 5.87%       | 93.95%         | 0.51%             |
| antirez Q4KExperts-F16             | 164.6     | 4.5726 | 0.0290             | 5.89%       | 93.94%         | 0.93%             |
| antirez mixed L37-42-Q4K (imatrix) | 97.6      | 5.8169 | 0.3605             | 21.15%      | 79.74%         | 0.41%             |
| antirez IQ2XXS (imatrix)           | 86.7      | 6.0808 | 0.4079             | 22.23%      | 78.15%         | 0.39%             |
| antirez IQ2XXS                     | 86.7      | 6.1518 | 0.4207             | 22.74%      | 77.92%         | 0.47%             |

### Official Benchmarks

DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on the benchmarks below despite using far fewer activated parameters, and remains competitive with leading proprietary models.

| Benchmark              | DeepSeek-V4-Flash-0731 | DeepSeek-V4-Flash (Preview) | DeepSeek-V4-Pro (Preview) | GLM-5.2 | Opus-4.8 |
| ---------------------- | :--------------------: | :-------------------------: | :-----------------------: | :-----: | :------: |
| Terminal Bench 2.1     |          82.7          |             61.8            |            72.1           |   81.0  |   85.0   |
| NL2Repo                |          54.2          |             39.4            |            38.5           |   48.9  |   69.7   |
| Cybergym               |          76.7          |             38.7            |            52.7           |    -    |   83.1   |
| DeepSWE                |          54.4          |             7.3             |            12.8           |   46.2  |   58.0   |
| Toolathlon-Verified    |          70.3          |             49.7            |            55.9           |   59.9  |   76.2   |
| Agents' Last Exam      |          25.2          |             15.8            |            16.5           |   23.8  |   25.7   |
| AutomationBench Public |          25.1          |             10.8            |            12.8           |   12.9  |   27.2   |
| DSBench-FullStack †    |          68.7          |             37.0            |            41.8           |   61.8  |   71.6   |
| DSBench-Hard †         |          59.6          |             25.8            |            31.1           |   54.5  |   71.7   |


# Large language model (LLMs) Tutorials

Discover the latest LLMs and learn how to run and fine-tune models locally for optimal performance with [Unsloth](https://github.com/unslothai/unsloth).

<table data-view="cards" data-search="true"><thead><tr><th></th><th data-hidden data-card-cover data-type="image">Cover image</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><a href="/docs/models/deepseek-v4.1-flash">DeepSeek-V4.1-Flash</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F4qwswsAZxmUaTEksSNJJ%2Fdeepseekv4.1.png?alt=media&amp;token=203263d6-e100-4c3f-9dd4-898391874d74">deepseekv4.1.png</a></td><td><a href="/docs/models/deepseek-v4.1-flash">DeepSeek-V4.1-Flash</a></td></tr><tr><td><a href="/docs/models/qwen3.8">Qwen3.8</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FSQJGLfV6lTdcvKUuBsjC%2Fqwen3.8%20logo.png?alt=media&amp;token=7942746b-409d-4064-8e66-221fceb920dd">qwen3.8 logo.png</a></td><td><a href="/docs/models/qwen3.8">Qwen3.8-27B</a></td></tr><tr><td><a href="/docs/models/qwen3.8-next">Qwen3.8-Flash-Next</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZLsomjRej3AKufXjHrF9%2Fqwen38%20flash%20new%20logo.png?alt=media&amp;token=adf61838-1b5e-4d2d-aa52-3ae4d9b728f1">qwen38 flash new logo.png</a></td><td><a href="/docs/models/qwen3.8-next">Qwen3.8-Flash-Next</a></td></tr><tr><td><a href="/docs/models/glm-5.3-flash">GLM-5.3-Flash</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FeHSdHWsGl4rQtUUgSkGj%2Fglm%205.3%20flash.png?alt=media&amp;token=acd21ab7-c5b5-4338-8b23-8d4cf16584f3">glm 5.3 flash.png</a></td><td><a href="/docs/models/glm-5.3-flash">GLM-5.3-Flash</a></td></tr><tr><td><a href="/docs/models/muse-spark-open">Meta Muse Spark Open</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQiRK6GPvPwgiesmclxOw%2Fmuse%20spark%20open%20logo.png?alt=media&amp;token=d4c4767d-b37b-4b67-859a-780245bd4f1c">muse spark open logo.png</a></td><td><a href="/docs/models/muse-spark-open">Meta Muse Spark Open</a></td></tr><tr><td><a href="/docs/models/qwen-image-2.1">Qwen-Image-2.1</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FjBLXH1oRP52poQcN1pOn%2Fqwen%20image%202.1%20logo.png?alt=media&amp;token=ab285b72-146a-4409-b9ee-327fb78cb127">qwen image 2.1 logo.png</a></td><td><a href="/docs/models/qwen-image-2.1">Qwen-Image-2.1</a></td></tr><tr><td><a href="/docs/models/embeddinggemma-2">EmbeddingGemma 2</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FazS5jiiTBR77ttEiqCrP%2Fembeddinggemma%202%20logo.png?alt=media&amp;token=0408c262-b5ea-49e5-a96a-c585a2a3fae6">embeddinggemma 2 logo.png</a></td><td><a href="/docs/models/embeddinggemma-2">EmbeddingGemma 2</a></td></tr><tr><td><a href="/docs/models/glm-5.3">GLM-5.3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FEOLfeTQcmdj4Ofo5jGhS%2Fglm%205.3%20logo.png?alt=media&amp;token=220f0d33-d5fc-4fc1-8152-126bd21f2cde">glm 5.3 logo.png</a></td><td><a href="/docs/models/glm-5.3">GLM-5.3</a></td></tr><tr><td><a href="/docs/models/muse-glimmer">Meta Muse Glimmer</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fnbdjif7yfvvH1j6aBmpS%2FMuse%20glimmer%20logo%20final.png?alt=media&amp;token=5031f9dc-c257-4488-95be-856e1ea31ec8">Muse glimmer logo final.png</a></td><td><a href="/docs/models/muse-glimmer">Meta Muse Glimmer</a></td></tr><tr><td><a href="/docs/models/deepseek-v4">DeepSeek-V4-Pro-0813</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FPa0WDeVwfBb1MT2pP2PW%2Fdeepseek%20v4%200813.png?alt=media&amp;token=f993598e-44ad-43ff-bfaf-90446167867f">deepseek v4 0813.png</a></td><td><a href="/docs/models/deepseek-v4">DeepSeek-V4-Flash</a></td></tr><tr><td><a href="/docs/models/deepseek-v4">DeepSeek-V4-0731</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fi3wY66ir0JOEOiCF9V1k%2Fdeepseekv4flash0731.png?alt=media&amp;token=a4d75067-0935-4720-ac2f-3ee752573cc4">deepseekv4flash0731.png</a></td><td><a href="/docs/models/deepseek-v4">DeepSeek-V4-Flash</a></td></tr><tr><td><a href="/docs/models/glm-5.3">GLM-5.3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FEOLfeTQcmdj4Ofo5jGhS%2Fglm%205.3%20logo.png?alt=media&amp;token=220f0d33-d5fc-4fc1-8152-126bd21f2cde">glm 5.3 logo.png</a></td><td><a href="/docs/models/glm-5.3">GLM-5.3</a></td></tr><tr><td><a href="/docs/models/kimi-k3">Kimi K3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwCzVjy7Q6EQLthkUuZqk%2Fkimik3%20logo.png?alt=media&amp;token=945dfb54-9c25-4539-9138-e23f841ca168">kimik3 logo.png</a></td><td><a href="/docs/models/kimi-k3">Kimi K3 </a></td></tr><tr><td><a href="/docs/models/glm-5.2">GLM-5.2</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmbYXj0v0p5zbeESPDYUr%2Fglm52.png?alt=media&amp;token=0cb4ae58-d249-403a-9fb5-72e9e93f8da9">glm52.png</a></td><td><a href="/docs/models/glm-5.2">GLM-5.2</a></td></tr><tr><td><a href="/docs/basics/nvfp4">Unsloth NVFP4</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJdza44RRT0sgZcx9B67a%2Fdynamic%20unsloth%20nvfp4.png?alt=media&amp;token=25d20d9d-bef4-43b5-b680-4ce1a37b4bd1">dynamic unsloth nvfp4.png</a></td><td><a href="/docs/basics/nvfp4">Unsloth Dynamic NVFP4</a></td></tr><tr><td><a href="/docs/models/mtp">MTP Guide</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fd1cFzqweTYCkjBF7aqlP%2Fmtp%20final%20guide.png?alt=media&amp;token=54932ff8-b0f5-409c-95cd-3887d7cb34c6">mtp final guide.png</a></td><td><a href="/docs/models/mtp">Run MTP Models</a></td></tr><tr><td><a href="/docs/models/deepseek-v4">DeepSeek-V4</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYCbdIeMsEQmDJoyii4tl%2Fdeepseek%20v4%20logo.png?alt=media&amp;token=0d1ec333-bcbd-4ecb-8891-5393a1c5bb0a">deepseek v4 logo.png</a></td><td><a href="/docs/models/deepseek-v4">DeepSeek-V4-Flash</a></td></tr><tr><td><a href="/docs/integrations/unsloth-start">Unsloth Start</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVB2vdala9hr9lGRgsf0B%2Funsloth%20start%20logo.png?alt=media&amp;token=add879d3-e53b-4480-ba7f-8d4759f60123">unsloth start logo.png</a></td><td><a href="/docs/integrations/unsloth-start">Unsloth Start</a></td></tr><tr><td><a href="/docs/models/diffusiongemma">DiffusionGemma</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FxAFkz46CwvWU7dYcntr8%2Fdiffusion%20gemma%20final%20logo.png?alt=media&amp;token=0e35a583-b147-4039-a2f6-b2446941c5a5">diffusion gemma final logo.png</a></td><td><a href="/docs/models/diffusiongemma">DiffusionGemma</a></td></tr><tr><td><a href="/docs/models/inkling">Inkling-Small</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FR3PZAuk8wn9xwW5blhc1%2Finkling%20small.png?alt=media&amp;token=dc7c879a-747e-4cea-86c3-f3f74b803274">inkling small.png</a></td><td><a href="/docs/models/inkling">Inkling</a></td></tr><tr><td><a href="/docs/models/inkling">Inkling</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLnhMXBfa2Ptpg4Gr1k6z%2Finklinggg.png?alt=media&amp;token=d3855fa7-50ff-4ef7-9802-480b8cb53469">inklinggg.png</a></td><td><a href="/docs/models/inkling">Inkling</a></td></tr><tr><td><a href="/docs/basics/amd">Unsloth for AMD</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FvyECRqXbIeD52Q4t9a04%2FAMD%20promo%20pic%201920.png?alt=media&amp;token=36563f7c-91ea-4b0f-a4b4-8c721e574808">AMD promo pic 1920.png</a></td><td><a href="/docs/basics/amd">AMD</a></td></tr><tr><td><a href="/docs/models/kimi-k2.7-code">Kimi K2.7 Code</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FYZCfnER3LjhmFPxeweOK%2Fkimi%20k27%20code%20font.png?alt=media&amp;token=3f01e9c9-8e26-4f34-99d7-d94d1f53a013">kimi k27 code font.png</a></td><td><a href="/docs/models/kimi-k2.7-code">Kimi K2.7 Code</a></td></tr><tr><td><a href="/docs/models/gemma-4/qat">Gemma 4 QAT</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fvs7qdmsXSjoutGnhvsQc%2Fgemma%204%20qat%20quant.png?alt=media&amp;token=a738247e-78de-4fd5-9504-2a1301efafbe">gemma 4 qat quant.png</a></td><td><a href="/docs/models/gemma-4/qat">Gemma 4 QAT</a></td></tr><tr><td><a href="/docs/models/minimax-m3">MiniMax M3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsaXyQq0XeFuudTK5ZEGw%2Fminimaxm3.png?alt=media&amp;token=4af28954-abd2-4163-a0fe-ed4e42e25d2f">minimaxm3.png</a></td><td><a href="/docs/models/minimax-m3">MiniMax M3</a></td></tr><tr><td><a href="/docs/models/qwen3.6#mtp-guide">Qwen3.6 MTP</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FERMxeauQ5uprqdxYmiI5%2Fqwen3.6%20mtp%20logo.png?alt=media&amp;token=f17e8b39-8566-4e45-b5fc-6e1491dab34e">qwen3.6 mtp logo.png</a></td><td><a href="/docs/models/qwen3.6#mtp-guide">Qwen3.6</a></td></tr><tr><td><a href="/docs/models/qwen3.6">Qwen3.6</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fdy7z03AUFzXHKqqOY7kC%2Fqwen3.6%20logo.png?alt=media&amp;token=a894c047-a9ea-4f9c-824a-be86ec81f54d">qwen3.6 logo.png</a></td><td><a href="/docs/models/qwen3.6">Qwen3.6</a></td></tr><tr><td><a href="/docs/models/gemma-4">Gemma 4</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkEjWOJqBWCtIN9Cg6CdI%2FGemma%204%20landscape.png?alt=media&amp;token=57d3f596-dae8-4eab-80e6-0847794ffc8d">Gemma 4 landscape.png</a></td><td><a href="/docs/models/gemma-4">Gemma 4</a></td></tr><tr><td><a href="/docs/basics/api">Unsloth API endpoint</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoRer1Vbuh9wSPTawz0ey%2Funsloth%20api%20long%20logo.png?alt=media&amp;token=860a8143-9d59-466b-abed-6ea7d67e51fd">unsloth api long logo.png</a></td><td><a href="/docs/basics/api">Unsloth API</a></td></tr><tr><td><a href="/docs/models/nemotron-3-ultra">NVIDIA Nemotron 3 Ultra</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F2wzlxjFzX3C3emfJguEC%2Fnemotron%20ultra%20logo.png?alt=media&amp;token=33378a3a-8984-424b-be3b-a072c4c34379">nemotron ultra logo.png</a></td><td></td></tr><tr><td><a href="/docs/models/nemotron-3-nano-omni">Nemotron 3 Nano Omni</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FutUcx0rRusdVoxK7pthE%2Fnvidia%20nemotron%203%20nano%20omni.png?alt=media&amp;token=9a0a6183-1592-4605-9226-210b0c433c98">nvidia nemotron 3 nano omni.png</a></td><td><a href="/docs/models/nemotron-3-nano-omni">NVIDIA Nemotron 3 Omni</a></td></tr><tr><td><a href="/docs/models/kimi-k2.6">Kimi K2.6</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJ0bVVb4T95GD9XONyMQh%2Fkimi%20k26.png?alt=media&amp;token=c69c24cd-45d5-4710-9c0f-d3a04ab7f07d">kimi k26.png</a></td><td><a href="/docs/models/kimi-k2.6">Kimi K2.6</a></td></tr><tr><td><a href="/docs/models/qwen3.5">Qwen3.5</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fvw6yRxJDCeBl1CIsQkki%2Fqwen35.png?alt=media&amp;token=28fe0357-351a-49e1-a176-bb21ecc8542a">qwen35.png</a></td><td><a href="/docs/models/qwen3.5">Qwen3.5</a></td></tr><tr><td><a href="/docs/models/mistral-3.5">Mistral 3.5</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmABUwHWrRwR7dBRUyL1h%2Fmistral3.5%20logo.png?alt=media&amp;token=1e304b9a-356f-46a1-aabe-454e56da5250">mistral3.5 logo.png</a></td><td><a href="/docs/models/mistral-3.5">Mistral 3.5</a></td></tr><tr><td><a href="/docs/models/glm-5.1">GLM-5.1</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FK69rPUGatLzCBK9uaqxU%2Fglm51%20logo.png?alt=media&amp;token=934ef701-0233-47fd-ad49-6c1a5959b684">glm51 logo.png</a></td><td><a href="/docs/models/glm-5.1">GLM-5.1</a></td></tr><tr><td><a href="/docs/models/ibm-granite-4.1">Granite 4.1</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpDixxClIZuLnbj5k5wZ2%2Fgranite4.1%20logo.png?alt=media&amp;token=2810fa43-0083-4382-b7e7-a903b631cab8">granite4.1 logo.png</a></td><td><a href="/docs/models/ibm-granite-4.1">IBM Granite 4.1</a></td></tr><tr><td><a href="/docs/models/tutorials/minimax-m27">MiniMax-2.7</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoWNqL9cQDvNtxEcwGfLY%2Fm27.png?alt=media&amp;token=aed5db72-4961-4089-b93a-01d9c4d19f0c">m27.png</a></td><td><a href="/docs/models/tutorials/minimax-m27">MiniMax-M2.7</a></td></tr><tr><td><a href="/docs/new/studio/start">Unsloth Studio: Run + train LLMs</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FyeXVnXerrNJ3cvMgZALH%2Funsloth%20studio%20logo%20only.png?alt=media&amp;token=1242ba64-b4d7-41a5-bd27-f480bd6c7e91">unsloth studio logo only.png</a></td><td><a href="/docs/new/studio/start">Get Started</a></td></tr><tr><td><a href="/docs/models/nemotron-3/nemotron-3-super">NVIDIA-Nemotron-3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FllPS7l6rpEr68mytlxXU%2Fnemotron%203%20logo.png?alt=media&amp;token=7bd05673-6b97-41c2-b657-530b7e6e4e3c">nemotron 3 logo.png</a></td><td><a href="/docs/models/nemotron-3/nemotron-3-super">NVIDIA Nemotron 3 Super</a></td></tr><tr><td><a href="/docs/models/tutorials/glm-5">GLM-5</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FINbIdckdXPbplF51fU92%2Fglm%205%20logo.png?alt=media&amp;token=53e4c484-c791-4ffe-a571-749e98d76b15">glm 5 logo.png</a></td><td><a href="/docs/models/tutorials/glm-5">GLM-5</a></td></tr><tr><td><a href="/docs/basics/claude-code">Claude Code</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FtwoMemhnRzpoFMGUQNHo%2Fclaude%20code%20white%20logo.png?alt=media&amp;token=e153fe82-bed1-4db9-b06d-9b5eb87a5212">claude code white logo.png</a></td><td><a href="/docs/basics/claude-code">Claude Code</a></td></tr><tr><td><a href="/docs/basics/codex">OpenAI Codex</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLxRuVHEBYywquhRGBYu0%2Fcodex%20only%20logo.png?alt=media&amp;token=7faa9cbd-090a-45dc-9d5e-acb1a9591f27">codex only logo.png</a></td><td><a href="/docs/basics/codex">OpenAI Codex</a></td></tr><tr><td><a href="/docs/models/tutorials/minimax-m25">MiniMax-2.5</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F0yrdjCKbV8qnqyTrQ1pZ%2Fminimax2.5%20logo.png?alt=media&amp;token=183839fe-6750-4c95-b058-c991ec8a5dec">minimax2.5 logo.png</a></td><td><a href="/docs/models/tutorials/minimax-m25">MiniMax-M2.5</a></td></tr><tr><td><a href="/docs/models/qwen3-coder-next">Qwen3-Coder-Next</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F47HGDuvMBPAkh4vcaGMg%2Fqwen3-coder-next%20logo.png?alt=media&amp;token=244ae539-fea4-40e8-9ee2-b6ec7fb44060">qwen3-coder-next logo.png</a></td><td><a href="/docs/models/qwen3-coder-next">Qwen3-Coder-Next</a></td></tr><tr><td><a href="/docs/models/tutorials/kimi-k2.5">Kimi K2.5</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FgcSsB0cPhjj8inDt1bqf%2Fkimi%20k25%20logo.png?alt=media&amp;token=19aec00a-7e0f-4980-b2b7-98b65a23123e">kimi k25 logo.png</a></td><td><a href="/docs/models/tutorials/kimi-k2.5">Kimi K2.5</a></td></tr><tr><td><a href="/docs/models/tutorials/deepseek-ocr-2">DeepSeek-OCR 2</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlHWXZZFIvvwS0LU56sF3%2Fdeepseekocr%202.png?alt=media&amp;token=c5b187f9-8aba-4739-9483-67f41e42924b">deepseekocr 2.png</a></td><td><a href="/docs/models/tutorials/deepseek-ocr-2">DeepSeek-OCR 2</a></td></tr><tr><td><a href="/docs/models/tutorials/glm-4.7-flash">GLM-4.7-Flash</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6PQZ23CoUdZs1EZCjtYn%2Fglm4.7flash.png?alt=media&amp;token=d3dc776e-ef3e-4eb3-ad4e-bf45e7b5745a">glm4.7flash.png</a></td><td><a href="/docs/models/tutorials/glm-4.7-flash">GLM-4.7-Flash</a></td></tr><tr><td><a href="/docs/models/tutorials/qwen-image-2512">Qwen-Image-2512</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FjOiEAPGTrPKs8g0cydOe%2FQWEN-IMAGE-2512.png?alt=media&amp;token=d9172aad-ebb2-4054-a5b2-b58c26c57290">QWEN-IMAGE-2512.png</a></td><td><a href="/docs/models/tutorials/qwen-image-2512">Qwen-Image-2512</a></td></tr><tr><td><a href="/docs/models/nemotron-3">Nemotron 3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F5AeVpAiKFQAgLpA7caad%2Fnemotron%20nano%203%20promo.png?alt=media&amp;token=e2f879a3-cf82-4253-8d62-9f0c0ac69375">nemotron nano 3 promo.png</a></td><td><a href="/docs/models/nemotron-3">NVIDIA Nemotron 3</a></td></tr><tr><td><a href="/docs/models/tutorials/functiongemma">FunctionGemma</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FSOnbpmZisRLEOOfZYhpF%2Ffunctiongemmaa.png?alt=media&amp;token=0fab5034-ed84-4199-a256-29da6fe1c164">functiongemmaa.png</a></td><td><a href="/docs/models/tutorials/functiongemma">FunctionGemma</a></td></tr><tr><td><a href="/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune">Qwen3-VL</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-72d99575af3aa1efc604082921964f68123c37a5%2Fqwen3-vl%20promo.png?alt=media">qwen3-vl promo.png</a></td><td><a href="/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune">Qwen3-VL</a></td></tr><tr><td><a href="/docs/models/tutorials/deepseek-ocr-how-to-run-and-fine-tune">DeepSeek-OCR</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4ef500f68903e0a800643c2be417d1fd879c9504%2Fdeepseek%20ocr%20logo.png?alt=media">deepseek ocr logo.png</a></td><td><a href="/docs/models/tutorials/deepseek-ocr-how-to-run-and-fine-tune">DeepSeek-OCR</a></td></tr><tr><td><a href="/docs/models/gpt-oss-how-to-run-and-fine-tune">Run gpt-oss</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1b3ddd70a1e3807dbaa1d16de8698737a89747ff%2Fgpt-oss%20image.png?alt=media">gpt-oss image.png</a></td><td><a href="/docs/models/gpt-oss-how-to-run-and-fine-tune">gpt-oss</a></td></tr><tr><td><a href="/docs/models/tutorials/ministral-3">Ministral 3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FdCdRGR8KYGpooEXYHJo2%2Fministral%203%20logo.png?alt=media&amp;token=79d76c89-b954-4dbd-8e1c-c1bdcf90b9a4">ministral 3 logo.png</a></td><td><a href="/docs/models/tutorials/ministral-3">Ministral 3</a></td></tr><tr><td><a href="/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally">Kimi K2 Thinking</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e4c3764508e91e7a98e2d98373bab8830935be13%2Fkimi%20k2%20thinking%20logo.png?alt=media">kimi k2 thinking logo.png</a></td><td><a href="/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally">Kimi K2 Thinking</a></td></tr><tr><td><a href="/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl">Vision Reinforcement Learning</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e57995d6adb92b887f1d48c80362720fc997a73b%2Fvision%20rl%20site.png?alt=media">vision rl site.png</a></td><td><a href="/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl">Vision RL</a></td></tr><tr><td><a href="/docs/models/tutorials/deepseek-v3.1-how-to-run-locally">DeepSeek-V3.1</a> Terminus</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-ff07aa43d4802064b63ed18cde1f6454c9048089%2Fdeepseek%20v3.1%20logo.png?alt=media">deepseek v3.1 logo.png</a></td><td><a href="/docs/models/tutorials/deepseek-v3.1-how-to-run-locally">DeepSeek-V3.1</a></td></tr><tr><td><a href="/docs/models/tutorials/qwen3-coder-how-to-run-locally">Qwen3 Coder</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-bee47a23ff14ed5c107787f0f4fc4182d13ede15%2Fqwen3-coder%201920.png?alt=media">qwen3-coder 1920.png</a></td><td><a href="/docs/models/tutorials/qwen3-coder-how-to-run-locally">Qwen3-Coder</a></td></tr><tr><td><a href="/docs/models/tutorials/magistral-how-to-run-and-fine-tune">Magistral 1.2</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-3ea62a89f90e8765b1866e5198bb5ec162d48b16%2Fmagistral%20center.png?alt=media">magistral center.png</a></td><td><a href="/docs/models/tutorials/magistral-how-to-run-and-fine-tune">Magistral</a></td></tr><tr><td><a href="/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune/gemma-3n-how-to-run-and-fine-tune">Gemma 3n</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-3edced131df4d2030d97ca9c48acdb654eedad4f%2FGemma%203%20text%20only.png?alt=media">Gemma 3 text only.png</a></td><td><a href="/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune/gemma-3n-how-to-run-and-fine-tune">Gemma 3n</a></td></tr><tr><td><a href="/docs/models/tutorials/qwen3-next"><strong>Qwen3-2507</strong></a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a3db5755f59934b300da12c4d9a64f13ad8de3f3%2Fqwen3-2507.png?alt=media">qwen3-2507.png</a></td><td><a href="/docs/models/tutorials/qwen3-next">Qwen3-Next</a></td></tr><tr><td><a href="/docs/models/tutorials/deepseek-r1-0528-how-to-run-locally">DeepSeek-R1-0528</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-398714041a6ff80d291b1ce0c7ef6541a8bff4aa%2Fdeepseek%20r1-0528.png?alt=media">deepseek r1-0528.png</a></td><td><a href="/docs/models/tutorials/deepseek-r1-0528-how-to-run-locally">DeepSeek-R1-0528</a></td></tr><tr><td><a href="/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally">Kimi K2</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-55660e8be362c073f7bb1ca4a95e05f4cd863469%2Fkimik2%20landcsape.png?alt=media">kimik2 landcsape.png</a></td><td><a href="/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally">Kimi K2 Thinking</a></td></tr><tr><td><a href="/docs/models/tutorials/devstral-how-to-run-and-fine-tune">Devstral 2507</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a7fe4299d9980650d28e332f463bbe90e0835dd0%2Fdevstral%20logo.png?alt=media">devstral logo.png</a></td><td><a href="/docs/models/tutorials/devstral-how-to-run-and-fine-tune">Devstral</a></td></tr><tr><td><a href="/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth">Fine-tune on Blackwell &#x26; RTX 50 GPUs</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-5d99af1e3e582338ee06c9ea4a7f7dee4ab3acd6%2Fnvidia-logo-white%20background.png?alt=media">nvidia-logo-white background.png</a></td><td><a href="/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth">Blackwell, RTX 50 and Unsloth</a></td></tr><tr><td><a href="/docs/basics/text-to-speech-tts-fine-tuning">TTS Fine-tuning</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-756c5d79b2e1b57a0f761e85dd7f16d04a5b9473%2Ftts%20finetuning%20landscape.png?alt=media">tts finetuning landscape.png</a></td><td><a href="/docs/basics/text-to-speech-tts-fine-tuning">Text-to-Speech Fine-tuning</a></td></tr><tr><td><a href="/docs/models/tutorials/qwen3-how-to-run-and-fine-tune">Qwen3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-183d5c53b411de2cea205cb9a898acd1da290919%2Fqwen3.png?alt=media">qwen3.png</a></td><td><a href="/docs/models/tutorials/qwen3-how-to-run-and-fine-tune">Qwen3</a></td></tr><tr><td><a href="/docs/models/tutorials/phi-4-reasoning-how-to-run-and-fine-tune">Phi-4 reasoning</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2d90fb4bb4655d3cc51bf91e1a10da073aceb424%2Fphi4%20reasoning2.png?alt=media">phi4 reasoning2.png</a></td><td><a href="/docs/models/tutorials/phi-4-reasoning-how-to-run-and-fine-tune">Phi-4 Reasoning</a></td></tr><tr><td><a href="https://github.com/unslothai/docs/blob/main/basics/unsloth-dynamic-2.0-ggufs">Dynamic 2.0 GGUFs</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-15a453d8ab8d1914bfe9cca781e8495a8f0cf98a%2Fdynamic%20v2%20with%20unsloth.png?alt=media">dynamic v2 with unsloth.png</a></td><td><a href="https://github.com/unslothai/docs/blob/main/basics/unsloth-dynamic-2.0-ggufs">https://github.com/unslothai/docs/blob/main/basics/unsloth-dynamic-2.0-ggufs</a></td></tr><tr><td><a href="/docs/models/tutorials/llama-4-how-to-run-and-fine-tune">Llama 4</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-ac8efc4f6648b524214c27042e35971b48083608%2Fllama%204%20only.png?alt=media">llama 4 only.png</a></td><td><a href="/docs/models/tutorials/llama-4-how-to-run-and-fine-tune">Llama 4</a></td></tr><tr><td><a href="/docs/models/tutorials/deepseek-v3-0324-how-to-run-locally">DeepSeek-V3-0324</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-286e3e541ecf3a096b5a0136e464a174d966f81c%2Fv30324.png?alt=media">v30324.png</a></td><td><a href="/docs/models/tutorials/deepseek-v3-0324-how-to-run-locally">DeepSeek-V3-0324</a></td></tr><tr><td><a href="/docs/models/tutorials/grok-2">Grok 2</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-9d1d6418829a2ea1f0fa3b14f44afc6519f8b991%2Fgrok%202%20logo.png?alt=media">grok 2 logo.png</a></td><td><a href="/docs/models/tutorials/grok-2">Grok 2</a></td></tr><tr><td><a href="/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune">Gemma 3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-55aa06d8eb6b3b831e1d78f9998ec4d8efec0a0b%2Fgemma%203%20logo.png?alt=media">gemma 3 logo.png</a></td><td><a href="/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune">Gemma 3</a></td></tr><tr><td><a href="/docs/models/tutorials/qwq-32b-how-to-run-effectively">QwQ-32B</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-674d877c9685ac88e94c87f04ffd40bd7cf92653%2Fqwq%20logo%20only.png?alt=media">qwq logo only.png</a></td><td><a href="/docs/models/tutorials/qwq-32b-how-to-run-effectively">QwQ-32B</a></td></tr><tr><td><a href="/docs/models/tutorials/deepseek-r1-how-to-run-locally">DeepSeek-R1</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a52bd4654a2f069a5c65b6f48a0729c0d4a08211%2Fdeepseek%20r1.png?alt=media">deepseek r1.png</a></td><td><a href="/docs/models/tutorials/deepseek-r1-how-to-run-locally">DeepSeek-R1</a></td></tr><tr><td><a href="/docs/get-started/reinforcement-learning-rl-guide">Reinforcement Learning (RL)</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-dd099bc8ea537c59f104a71021f28f1458303c23%2Frl%20guide%20new.png?alt=media">rl guide new.png</a></td><td><a href="/docs/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo">Tutorial: GRPO Training</a></td></tr><tr><td><a href="https://www.unsloth.ai/blog/mistral-small-3.1">Mistral Small 3.1</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-54ce6230652c7a5a4421019c9e78feb0652f9def%2Fmistral%20small%203.1.png?alt=media">mistral small 3.1.png</a></td><td><a href="https://www.unsloth.ai/blog/mistral-small-3.1">https://www.unsloth.ai/blog/mistral-small-3.1</a></td></tr><tr><td><a href="/docs/get-started/fine-tuning-llms-guide/tutorial-how-to-finetune-llama-3-and-use-in-ollama">Llama 3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-363677e7a2a06bebbf47651776b1982c85efcc95%2Fllama%203logo.png?alt=media">llama 3logo.png</a></td><td><a href="/docs/get-started/fine-tuning-llms-guide/tutorial-how-to-finetune-llama-3-and-use-in-ollama">Tutorial: Finetune Llama-3 and Use In Ollama</a></td></tr><tr><td><a href="/docs/basics/vision-fine-tuning">Vision Fine-tuning</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-7920bd48ca2e5853a4da3260e2d6383ebb3f4f77%2Fllama_3.2_vision_large_rectangle_jPUNULJrVe5O4AvDDWO1M.webp?alt=media">llama_3.2_vision_large_rectangle_jPUNULJrVe5O4AvDDWO1M.webp</a></td><td><a href="/docs/basics/vision-fine-tuning">Vision Fine-tuning</a></td></tr><tr><td><a href="/docs/basics/continued-pretraining">Continued Pretraining</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-67372072ccfb8f3754ebe061457d2b113ed36ccc%2Fcontinued_pretraining_just_graph_HC0ALBypfCXyUUXClYPiN.webp?alt=media">continued_pretraining_just_graph_HC0ALBypfCXyUUXClYPiN.webp</a></td><td><a href="/docs/basics/continued-pretraining">Continued Pretraining</a></td></tr><tr><td><a href="https://unsloth.ai/blog/llama3-3">Llama 3.3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-5707b8f8a8863be773aa07f8733e1cde0dd6bcc4%2Fllama_3.3_website_9hQURhj6KfZ7EnBRaKbiu.webp?alt=media">llama_3.3_website_9hQURhj6KfZ7EnBRaKbiu.webp</a></td><td><a href="https://unsloth.ai/blog/llama3-3">https://unsloth.ai/blog/llama3-3</a></td></tr><tr><td><a href="https://unsloth.ai/blog/gemma2">Gemma 2</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-5522df8efc21f7ef1b56d50497cddc692741e2ce%2Fgemma_2_long_OKsRGiTB8vrcIyXNWdgMw.avif?alt=media">gemma_2_long_OKsRGiTB8vrcIyXNWdgMw.avif</a></td><td><a href="https://unsloth.ai/blog/gemma2">https://unsloth.ai/blog/gemma2</a></td></tr><tr><td><a href="https://unsloth.ai/blog/phi3">Phi-3</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-eb6cb4c699a505b419f6b71c1ead33898891e02f%2Fphi3_unsloth_ynBY7FG3NTjIbS11ozN_g.webp?alt=media">phi3_unsloth_ynBY7FG3NTjIbS11ozN_g.webp</a></td><td><a href="https://unsloth.ai/blog/phi3">https://unsloth.ai/blog/phi3</a></td></tr></tbody></table>


# Qwen3 - How to Run & Fine-tune

Learn to run & fine-tune Qwen3 locally with Unsloth + our Dynamic 2.0 quants

Qwen's new Qwen3 models deliver state-of-the-art advancements in reasoning, instruction-following, agent capabilities, and multilingual support.

{% hint style="success" %}
**NEW!** Qwen3 got an update in July 2025. Run & fine-tune the latest model: [**Qwen-2507**](/docs/models/tutorials/qwen3-next)
{% endhint %}

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized Qwen LLMs with minimal accuracy loss.

We also uploaded Qwen3 with native 128K context length. Qwen achieves this by using YaRN to extend its original 40K window to 128K.

[Unsloth](https://github.com/unslothai/unsloth) also now supports fine-tuning and [Reinforcement Learning (RL)](/docs/get-started/reinforcement-learning-rl-guide) of Qwen3 and Qwen3 MOE models — 2x faster, with 70% less VRAM, and 8x longer context lengths. Fine-tune Qwen3 (14B) for free using our [Colab notebook.](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(14B\)-Reasoning-Conversational.ipynb)

<a href="#running-qwen3" class="button primary">Running Qwen3 Tutorial</a> <a href="#fine-tuning-qwen3-with-unsloth" class="button secondary">Fine-tuning Qwen3</a>

#### **Qwen3 - Unsloth Dynamic 2.0** with optimal configs:

| Dynamic 2.0 GGUF (to run)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | 128K Context GGUF                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Dynamic 4-bit Safetensor (to finetune/deploy)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <ul><li><a href="https://huggingface.co/unsloth/Qwen3-0.6B-GGUF">0.6B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-1.7B-GGUF">1.7B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-4B-GGUF">4B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-8B-GGUF">8B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-14B-GGUF">14B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF">30B-A3B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-32B-GGUF">32B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-235B-A22B-GGUF">235B-A22B</a></li></ul> | <ul><li><a href="https://huggingface.co/unsloth/Qwen3-4B-128K-GGUF">4B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-8B-128K-GGUF">8B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-14B-128K-GGUF">14B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-30B-A3B-128K-GGUF">30B-A3B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-32B-128K-GGUF">32B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-235B-A22B-128K-GGUF">235B-A22B</a></li></ul> | <ul><li><a href="https://huggingface.co/unsloth/Qwen3-0.6B-unsloth-bnb-4bit">0.6B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-1.7B-unsloth-bnb-4bit">1.7B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-4B-unsloth-bnb-4bit">4B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-8B-unsloth-bnb-4bit">8B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-14B-unsloth-bnb-4bit">14B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-30B-A3B-bnb-4bit">30B-A3B</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-32B-unsloth-bnb-4bit">32B</a></li></ul> |

## 🖥️ **Running Qwen3**

To achieve inference speeds of 6+ tokens per second, we recommend your available memory should match or exceed the size of the model you’re using. For example, a 30GB 1-bit quantized model requires at least 150GB of memory. The Q2\_K\_XL quant, which is 180GB, will require at least **180GB of unified memory** (VRAM + RAM) or **180GB of RAM** for optimal performance.

**NOTE:** It’s possible to run the model with **less total memory** than its size (i.e., less VRAM, less RAM, or a lower combined total). However, this will result in slower inference speeds. Sufficient memory is only required if you want to maximize throughput and achieve the fastest inference times.

### :gear: Official Recommended Settings

According to Qwen, these are the recommended settings for inference:

| Non-Thinking Mode Settings:                                            | Thinking Mode Settings:                                           |
| ---------------------------------------------------------------------- | ----------------------------------------------------------------- |
| <mark style="background-color:blue;">**Temperature = 0.7**</mark>      | <mark style="background-color:blue;">**Temperature = 0.6**</mark> |
| Min\_P = 0.0 (optional, but 0.01 works well, llama.cpp default is 0.1) | Min\_P = 0.0                                                      |
| Top\_P = 0.8                                                           | Top\_P = 0.95                                                     |
| TopK = 20                                                              | TopK = 20                                                         |

**Chat template/prompt format:**

{% code overflow="wrap" %}

```
<|im_start|>user\nWhat is 2+2?<|im_end|>\n<|im_start|>assistant\n
```

{% endcode %}

{% hint style="success" %}
For NON thinking mode, we purposely enclose \<think> and \</think> with nothing:
{% endhint %}

{% code overflow="wrap" %}

```
<|im_start|>user\nWhat is 2+2?<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n
```

{% endcode %}

{% hint style="warning" %}
**For Thinking-mode, DO NOT use greedy decoding**, as it can lead to performance degradation and endless repetitions.
{% endhint %}

### Switching Between Thinking and Non-Thinking Mode

Qwen3 models come with built-in "thinking mode" to boost reasoning and improve response quality - similar to how [QwQ-32B](/docs/models/tutorials/qwq-32b-how-to-run-effectively) worked. Instructions for switching will differ depending on the inference engine you're using so ensure you use the correct instructions.

#### Instructions for llama.cpp and Ollama:

You can add `/think` and `/no_think` to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi-turn conversations.

Here is an example of multi-turn conversation:

```
> Who are you /no_think

<think>

</think>

I am Qwen, a large-scale language model developed by Alibaba Cloud. [...]

> How many 'r's are in 'strawberries'? /think

<think>
Okay, let's see. The user is asking how many times the letter 'r' appears in the word "strawberries". [...]
</think>

The word strawberries contains 3 instances of the letter r. [...]
```

#### Instructions for transformers and vLLM:

**Thinking mode:**

`enable_thinking=True`

By default, Qwen3 has thinking enabled. When you call `tokenizer.apply_chat_template`, you **don’t need to set anything manually.**

```python
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True  # Default is True
)
```

In thinking mode, the model will generate an extra `<think>...</think>` block before the final answer — this lets it "plan" and sharpen its responses.

**Non-thinking mode:**

`enable_thinking=False`

Enabling non-thinking will make Qwen3 will skip all the thinking steps and behave like a normal LLM.

```python
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False  # Disables thinking mode
)
```

This mode will provide final responses directly — no `<think>` blocks, no chain-of-thought.

### 🦙 Ollama: Run Qwen3 Tutorial

1. Install `ollama` if you haven't already! You can only run models up to 32B in size. To run the full 235B-A22B model, [see here](#running-qwen3-235b-a22b).

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```bash
ollama run hf.co/unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL
```

3. To disable thinking, use (or you can set it in the system prompt):

```
>>> Write your prompt here /nothink
```

{% hint style="warning" %}
If you're experiencing any looping, Ollama might have set your context length window to 2,048 or so. If this is the case, bump it up to 32,000 and see if the issue still persists.
{% endhint %}

### 📖 Llama.cpp: Run Qwen3 Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-14B-GGUF",
    local_dir = "unsloth/Qwen3-14B-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

3. Run the model and try any prompt.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3-14B-GGUF/Qwen3-14B-UD-Q2_K_XL.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --seed 3407 \
    --prio 3 \
    --temp 0.6 \
    --min-p 0.0 \
    --top-p 0.95 \
    --top-k 20 \
    -no-cnv
```

To disable thinking, use (or you can set it in the system prompt):

```
>>> Write your prompt here /nothink
```

### Running Qwen3-235B-A22B

For Qwen3-235B-A22B, we will specifically use Llama.cpp for optimized inference and a plethora of options.

1. We're following similar steps to above however this time we'll also need to perform extra steps because the model is so big.
2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions..

   ```python
   # !pip install huggingface_hub hf_transfer
   import os
   os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
   from huggingface_hub import snapshot_download
   snapshot_download(
       repo_id = "unsloth/Qwen3-235B-A22B-GGUF",
       local_dir = "unsloth/Qwen3-235B-A22B-GGUF",
       allow_patterns = ["*UD-Q2_K_XL*"],
   )
   ```
3. Run the model and try any prompt.
4. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% hint style="success" %}
Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.
{% endhint %}

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3-235B-A22B-GGUF/Qwen3-235B-A22B-UD-Q2_K_XL.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --seed 3407 \
    --prio 3 \
    --temp 0.6 \
    --min-p 0.0 \
    --top-p 0.95 \
    --top-k 20 \
    -no-cnv \
    --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n"
```

{% endcode %}

## 🦥 Fine-tuning Qwen3 with Unsloth

Unsloth makes Qwen3 fine-tuning 2x faster, use 70% less VRAM and supports 8x longer context lengths. Qwen3 (14B) fits comfortably in a Google Colab 16GB VRAM Tesla T4 GPU.

Because Qwen3 supports both reasoning and non-reasoning, you can fine-tune it with a non-reasoning dataset, but this may affect its reasoning ability. If you want to maintain its reasoning capabilities (optional), you can use a mix of direct answers and chain-of-thought examples. Use <mark style="background-color:green;">75% reasoning</mark> and <mark style="background-color:green;">25% non-reasoning</mark> in your dataset to make the model retain its reasoning capabilities.

Our Conversational notebook uses a combo of 75% NVIDIA’s open-math-reasoning dataset and 25% Maxime’s FineTome dataset (non-reasoning). Here's free Unsloth Colab notebooks to fine-tune Qwen3:

* [Qwen3 (14B) Reasoning + Conversational notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(14B\)-Reasoning-Conversational.ipynb) (recommended)
* [**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-GRPO.ipynb) **- Advanced GRPO LoRA**
* [Qwen3 (14B) Alpaca notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(14B\)-Alpaca.ipynb) (for Base models)

If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth:

```
pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo
```

### Qwen3 MOE models fine-tuning

Fine-tuning support includes our new 2026 [Faster MOE ](/docs/basics/faster-moe)update: 30B-A3B and 235B-A22B. Qwen3-30B-A3B works on just 17.5GB VRAM with Unsloth. On fine-tuning MoE's - it's probably not a good idea to fine-tune the router layer so we disabled it by default.

The 30B-A3B fits in 17.5GB VRAM, but you may lack RAM or disk space since the full 16-bit model must be downloaded and converted to 4-bit on the fly for QLoRA fine-tuning. This is due to issues importing 4-bit BnB MOE models directly. This only affects MOE models.

```python
from unsloth import FastModel
import torch
model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3-30B-A3B",
    max_seq_length = 2048, # Choose any for long context!
    load_in_4bit = True,  # 4 bit quantization to reduce memory
    load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory
    full_finetuning = False, # [NEW!] We have full finetuning now!
    # token = "hf_...", # use one if using gated models
)
```

### Notebook Guide:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1cfff4a8a728737598e1432529da745d58e71248%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

To use the notebooks, just click Runtime, then Run all. You can change settings in the notebook to whatever you desire. We have set them automatically by default. Change model name to whatever you like by matching it with model's name on Hugging Face e.g. 'unsloth/Qwen3-8B' or 'unsloth/Qwen3-0.6B-unsloth-bnb-4bit'.

There are other settings which you can toggle:

* **`max_seq_length = 2048`** – Controls context length. While Qwen3 supports 40960, we recommend 2048 for testing. Unsloth enables 8× longer context fine-tuning.
* **`load_in_4bit = True`** – Enables 4-bit quantization, reducing memory use 4× for fine-tuning on 16GB GPUs.
* For **full-finetuning** - set `full_finetuning = True` and **8-bit finetuning** - set `load_in_8bit = True`

If you'd like to read a full end-to-end guide on how to use Unsloth notebooks for fine-tuning or just learn about fine-tuning, creating [datasets](/docs/get-started/fine-tuning-llms-guide/datasets-guide) etc., view our [complete guide here](/docs/get-started/fine-tuning-llms-guide):

{% content-ref url="/pages/nw2c1elNySGBBav8WP9B" %}
[Fine-tuning Guide](/docs/get-started/fine-tuning-llms-guide)
{% endcontent-ref %}

{% content-ref url="/pages/XgcpRfamZHmHnRHBnVE4" %}
[Datasets Guide](/docs/get-started/fine-tuning-llms-guide/datasets-guide)
{% endcontent-ref %}

### GRPO with Qwen3

We made a new advanced GRPO notebook for fine-tuning Qwen3. Learn to use our new proximity-based reward function (closer answers = rewarded) and Hugging Face's Open-R1 math dataset.\
Unsloth now also has better evaluations and uses the latest version of vLLM.

[**Qwen3 (4B)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-GRPO.ipynb) **notebook - Advanced GRPO LoRA**

Learn about:

* Enabling reasoning in Qwen3 (Base)+ guiding it to do a specific task
* Pre-finetuning to bypass GRPO's tendency to learn formatting
* Improved evaluation accuracy via new regex matching
* Custom GRPO templates beyond just 'think' e.g. \<start\_working\_out>\</end\_working\_out>
* Proximity-based scoring: better answers earn more points (e.g., predicting 9 when the answer is 10) and outliers are penalized

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-be34c101c627020c7a6cfb6cd249f2462587d235%2Fqwen33%20mascot.png?alt=media" alt=""><figcaption></figcaption></figure>


# Qwen3-VL: How to Run Guide

Learn to fine-tune and run Qwen3-VL locally with Unsloth.

Qwen3-VL is Qwen’s new vision models with **instruct** and **thinking** versions. The 2B, 4B, 8B and 32B models are dense, while 30B and 235B are MoE. The 235B thinking LLM delivers SOTA vision and coding performance rivaling GPT-5 (high) and Gemini 2.5 Pro.\
\
Qwen3-VL has vision, video and OCR capabilities as well as 256K context (can be extended to 1M).\
\
[Unsloth](https://github.com/unslothai/unsloth) supports **Qwen3-VL fine-tuning and** [**RL**](/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl). Train Qwen3-VL (8B) for free with our [notebooks](#fine-tuning-qwen3-vl).

<a href="#running-qwen3-vl" class="button primary">Running Qwen3-VL</a><a href="#fine-tuning-qwen3-vl" class="button secondary">Fine-tuning Qwen3-VL</a>

## 🖥️ **Running Qwen3-VL**

To run the model in llama.cpp, vLLM, Ollama etc., here are the recommended settings:

### :gear: Recommended Settings

Qwen recommends these settings for both models (they're a bit different for Instruct vs Thinking):

| Instruct Settings:                                                       | Thinking Settings:                                                       |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------ |
| <mark style="background-color:blue;">**Temperature = 0.7**</mark>        | <mark style="background-color:blue;">**Temperature = 1.0**</mark>        |
| <mark style="background-color:yellow;">**Top\_P = 0.8**</mark>           | <mark style="background-color:yellow;">**Top\_P = 0.95**</mark>          |
| <mark style="background-color:green;">**presence\_penalty = 1.5**</mark> | <mark style="background-color:green;">**presence\_penalty = 0.0**</mark> |
| Output Length = 32768 (up to 256K)                                       | Output Length = 40960 (up to 256K)                                       |
| Top\_K = 20                                                              | Top\_K = 20                                                              |

Qwen3-VL also used the below settings for their benchmarking numbers, as mentioned [on GitHub](https://github.com/QwenLM/Qwen3-VL/tree/main?tab=readme-ov-file#generation-hyperparameters).

{% columns %}
{% column %}
Instruct Settings:

```bash
export greedy='false'
export seed=3407
export top_p=0.8
export top_k=20
export temperature=0.7
export repetition_penalty=1.0
export presence_penalty=1.5
export out_seq_length=32768
```

{% endcolumn %}

{% column %}
Thinking Settings:

```bash
export greedy='false'
export seed=1234
export top_p=0.95
export top_k=20
export temperature=1.0
export repetition_penalty=1.0
export presence_penalty=0.0
export out_seq_length=40960
```

{% endcolumn %}
{% endcolumns %}

### :bug:Chat template bug fixes

At Unsloth, we care about accuracy the most, so we investigated why after the 2nd turn of running the Thinking models, llama.cpp would break, as seen below:

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-37356b40688b10a85c927e1d432739a15bb33682%2Fimage.webp?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}
The error code:

```
terminate called after throwing an instance of 'std::runtime_error'
  what():  Value is not callable: null at row 63, column 78:
            {%- if '</think>' in content %}
                {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\n').split('<think>')|last).lstrip('\n') %}
                                                                             ^
```

{% endcolumn %}
{% endcolumns %}

We have successfully fixed the Thinking chat template for the VL models so we re-uploaded all Thinking quants and Unsloth's quants. They should now all work after the 2nd conversation - **other quants will fail to load after the 2nd conversation.**

### **Qwen3-VL Unsloth uploads**:

Qwen3-VL is now supported for GGUFs by llama.cpp as of 30th October 2025, so you can run them locally!

| Dynamic GGUFs (to run)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | 4-bit BnB Unsloth Dynamic                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | 16-bit full-precision                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <ul><li><a href="https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-GGUF">2B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-GGUF">2B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF">4B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-GGUF">4B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF">8B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-GGUF">8B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Instruct-GGUF">30B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Thinking-GGUF">30B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-32B-Instruct-GGUF">32B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-32B-Thinking-GGUF">32B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Instruct-GGUF">235B-A22B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Thinking-GGUF">235B-A22B-Thinking</a></li></ul> | <ul><li><a href="https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct-unsloth-bnb-4bit">2B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-2B-Thinking-unsloth-bnb-4bit">2B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-unsloth-bnb-4bit">4B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking-unsloth-bnb-4bit">4B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit">8B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-unsloth-bnb-4bit">8B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-32B-Instruct-unsloth-bnb-4bit">32B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-32B-Thinking-unsloth-bnb-4bit">32B-Thinking</a></li></ul> | <ul><li><a href="https://huggingface.co/unsloth/Qwen3-VL-2B-Instruct">2B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct">4B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-4B-Thinking">4B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct">8B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking">8B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Instruct">30B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-30B-A3B-Thinking">30B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-32B-Instruct">32B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-32B-Thinking">32B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Thinking">235B-A22B-Thinking</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-VL-235B-A22B-Instruct">235B-A22B-Instruct</a></li></ul> |

### 📖 Llama.cpp: Run Qwen3-VL Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. **Let's first get an image!** You can also upload images as well. We shall use <https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/unsloth%20made%20with%20love.png>, which is just our mini logo showing how finetunes are made with Unsloth:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-9bf7ec93680f889d7602e5f56a8d677d6a58ae6a%2Funsloth%20made%20with%20love.png?alt=media" alt="" width="188"><figcaption></figcaption></figure>

3. Let's download this image

{% code overflow="wrap" %}

```bash
wget https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/unsloth%20made%20with%20love.png -O unsloth.png
```

{% endcode %}

4. Let's get the 2nd image at <https://files.worldwildlife.org/wwfcmsprod/images/Sloth_Sitting_iStock_3_12_2014/story_full_width/8l7pbjmj29_iStock_000011145477Large_mini__1_.jpg>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4b30cc86b2c75edf95ee1ec6fe0c51fb30afd6c0%2F8l7pbjmj29_iStock_000011145477Large_mini__1_.jpg?alt=media" alt="" width="188"><figcaption></figcaption></figure>

{% code overflow="wrap" %}

```bash
wget https://files.worldwildlife.org/wwfcmsprod/images/Sloth_Sitting_iStock_3_12_2014/story_full_width/8l7pbjmj29_iStock_000011145477Large_mini__1_.jpg -O picture.png
```

{% endcode %}

5. Then, let's use llama.cpp's auto model downloading feature, try this for the 8B Instruct model:

```bash
./llama.cpp/llama-mtmd-cli \
    -hf unsloth/Qwen3-VL-8B-Instruct-GGUF:UD-Q4_K_XL \
    --n-gpu-layers 99 \
    --jinja \
    --top-p 0.8 \
    --top-k 20 \
    --temp 0.7 \
    --min-p 0.0 \
    --flash-attn on \
    --presence-penalty 1.5 \
    --ctx-size 8192
```

6. Once in, you will see the below screen:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-636dfd126430a8a8c91ef6d248b007daa34561c5%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

7. Load up the image via `/image PATH` ie `/image unsloth.png` then press ENTER

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-7525265b8ef19c7fd17cca64d1b64ffe1959c2d1%2Fimage.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

8. When you hit ENTER, it'll say "unsloth.png image loaded"

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2c996efe3373ae7f05bfec4d214524768624a6a8%2Fimage.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

9. Now let's ask a question like "What is this image?":

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-62bd79e094c7daad6a8f021194aa0e67ef96f9a5%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

10. Now load in picture 2 via `/image picture.png` then hit ENTER and ask "What is this image?"

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-317cc2c7e41765ff466d357d14d506115f3262b6%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

11. And finally let's ask how are both images are related (it works!)

{% code overflow="wrap" %}

```
The two images are directly related because they both feature the **tree sloth**, which is the central subject of the "made with unsloth" project.

- The first image is the **official logo** for the "made with unsloth" project. It features a stylized, cartoonish tree sloth character inside a green circle, with the text "made with unsloth" next to it. This is the visual identity of the project.
- The second image is a **photograph** of a real tree sloth in its natural habitat. This photo captures the animal's physical appearance and behavior in the wild.

The relationship between the two images is that the logo (image 1) is a digital representation or symbol used to promote the "made with unsloth" project, while the photograph (image 2) is a real-world depiction of the actual tree sloth. The project likely uses the character from the logo as an icon or mascot, and the photograph serves to illustrate what the tree sloth looks like in its natural environment.
```

{% endcode %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e323226293156ac17708836c635c6df3ab2b9ca3%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

12. You can also download the model via (after installing `pip install huggingface_hub hf_transfer` ) HuggingFace's `snapshot_download` which is useful for large model downloads, **since llama.cpp's auto downloader might lag.** You can choose Q4\_K\_M, or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id   = "unsloth/Qwen3-VL-8B-Instruct-GGUF", # Or "unsloth/Qwen3-VL-8B-Thinking-GGUF"
    local_dir = "unsloth/Qwen3-VL-8B-Instruct-GGUF", # Or "unsloth/Qwen3-VL-8B-Thinking-GGUF"
    allow_patterns = ["*UD-Q4_K_XL*", "*mmproj-F16*"],
)
```

13. Run the model and try any prompt. **For Instruct:**

```bash
./llama.cpp/llama-mtmd-cli \
    --model unsloth/Qwen3-VL-8B-Instruct-GGUF/Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3-VL-8B-Instruct-GGUF/mmproj-F16.gguf \
    --n-gpu-layers 99 \
    --jinja \
    --top-p 0.8 \
    --top-k 20 \
    --temp 0.7 \
    --min-p 0.0 \
    --flash-attn on \
    --presence-penalty 1.5 \
    --ctx-size 8192
```

14. **For Thinking**:

```bash
./llama.cpp/llama-mtmd-cli \
    --model unsloth/Qwen3-VL-8B-Thinking-GGUF/Qwen3-VL-8B-Thinking-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3-VL-8B-Thinking-GGUF/mmproj-F16.gguf \
    --n-gpu-layers 99 \
    --jinja \
    --top-p 0.95 \
    --top-k 20 \
    --temp 1.0 \
    --min-p 0.0 \
    --flash-attn on \
    --presence-penalty 0.0 \
    --ctx-size 8192
```

### :magic\_wand:Running Qwen3-VL-235B-A22B and Qwen3-VL-30B-A3B

For Qwen3-VL-235B-A22B, we will use llama.cpp for optimized inference and a plethora of options.

1. We're following similar steps to above however this time we'll also need to perform extra steps because the model is so big.
2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions..

   ```python
   # !pip install huggingface_hub hf_transfer
   import os
   os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
   from huggingface_hub import snapshot_download
   snapshot_download(
       repo_id = "unsloth/Qwen3-VL-235B-A22B-Instruct-GGUF",
       local_dir = "unsloth/Qwen3-VL-235B-A22B-Instruct-GGUF",
       allow_patterns = ["*UD-Q2_K_XL*", "*mmproj-F16*"],
   )
   ```
3. Run the model and try a prompt. Set the correct parameters for Thinking vs. Instruct.

**Instruct:**

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-mtmd-cli \
    --model unsloth/Qwen3-VL-235B-A22B-Instruct-GGUF/UD-Q2_K_XL/Qwen3-VL-235B-A22B-Instruct-UD-Q2_K_XL-00001-of-00002.gguf \
    --mmproj unsloth/Qwen3-VL-235B-A22B-Instruct-GGUF/mmproj-F16.gguf
    --jinja \
    --top-p 0.8 \
    --top-k 20 \
    --temp 0.7 \
    --min-p 0.0 \
    --flash-attn on \
    --presence-penalty 1.5 \
    --ctx-size 8192 \
```

{% endcode %}

**Thinking:**

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-mtmd-cli \
    --model unsloth/Qwen3-VL-235B-A22B-Thinking-GGUF/UD-Q2_K_XL/Qwen3-VL-235B-A22B-Thinking-UD-Q2_K_XL-00001-of-00002.gguf \
    --mmproj unsloth/Qwen3-VL-235B-A22B-Thinking-GGUF/mmproj-F16.gguf \
    --n-gpu-layers 99 \
    --jinja \
    --top-p 0.95 \
    --top-k 20 \
    --temp 1.0 \
    --min-p 0.0 \
    --flash-attn on \
    --presence-penalty 0.0 \
    --ctx-size 8192 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endcode %}

4. Edit, `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% hint style="success" %}
**Use `--fit on` introduced 15th Dec 2025 for maximum usage of your GPU and CPU.**

Optionally, use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.
{% endhint %}

### 🐋 Docker: Run Qwen3-VL

If you already have Docker desktop, to run Unsloth's models from Hugging Face, run the command below and you're done:

```bash
docker model pull hf.co/unsloth/Qwen3-VL-8B-Instruct-GGUF:UD-Q4_K_XL
```

Or you can run Docker's uploaded Qwen3-VL models:

```bash
docker model run ai/qwen3-vl
```

## 🦥 **Fine-tuning Qwen3-VL**

Unsloth supports fine-tuning and reinforcement learning (RL) Qwen3-VL including the larger 32B and 235B models. This includes support for fine-tuning for video and object detection. As usual, Unsloth makes Qwen3-VL models train 1.7x faster with 60% less VRAM and 8x longer context lengths with no accuracy degradation.\
\
We made two Qwen3-VL (8B) training notebooks which you can train free on Colab:

* [Normal SFT fine-tuning notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision.ipynb)
* [GRPO/GSPO RL notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_\(8B\)-Vision-GRPO.ipynb)

{% hint style="success" %}
**Saving Qwen3-VL to GGUF now works as llama.cpp just supported it!**

If you want to use any other Qwen3-VL model, just change the 8B model to the 2B, 32B etc. one.
{% endhint %}

The goal of the GRPO notebook is to make a vision language model solve maths problems via RL given an image input like below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-fe1591d4378d19fa5115f61680d60356846807f5%2Four_new_3_datasets.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

This Qwen3-VL support also integrates our latest update for even more memory efficient + faster RL including our [Standby feature](/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl#unsloth-standby), which uniquely limits speed degradation compared to other implementations. You can read more about how to train vision LLMs with RL with our [VLM GRPO guide](/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl).

### Multi-image training

In order to fine-tune or train Qwen3-VL with multi-images the most straightforward change is to swap

```python
ds_converted = ds.map(
    convert_to_conversation,
)
```

with:

```python
ds_converted = [convert_to_converation(sample) for sample in dataset]
```

Using map kicks in dataset standardization and arrow processing rules which can be strict and more complicated to define.


# Qwen3-2507: Run Locally Guide

Run Qwen3-30B-A3B-2507 and 235B-A22B Thinking and Instruct versions locally on your device!

Qwen released 2507 (July 2025) updates for their [Qwen3](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune) 4B, 30B and 235B models, introducing both "thinking" and "non-thinking" variants. The non-thinking '**Qwen3-30B-A3B-Instruct-2507**' and '**Qwen3-235B-A22B-Instruct-2507'** features a 256K context window, improved instruction following, multilingual capabilities and alignment.

The thinking models '**Qwen3-30B-A3B-Thinking-2507**' and '**Qwen3-235B-A22B-Thinking-2507**' excel at reasoning, with the 235B achieving SOTA results in logic, math, science, coding, and advanced academic tasks.

[Unsloth](https://github.com/unslothai/unsloth) also now supports fine-tuning and [Reinforcement Learning (RL)](/docs/get-started/reinforcement-learning-rl-guide) of Qwen3-2507 models — 2x faster, with 70% less VRAM, and 8x longer context lengths

<a href="#run-qwen3-30b-a3b-2507-tutorials" class="button secondary">Run 30B-A3B</a><a href="/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-2507#run-qwen3-235b-a22b-2507-tutorials" class="button secondary">Run 235B-A22B</a><a href="#fine-tuning-qwen3-2507-with-unsloth" class="button secondary">Fine-tune Qwen3-2507</a>

**Unsloth** [**Dynamic 2.0**](/docs/basics/dynamic-3.0-ggufs) **GGUFs:**

| Model                    | GGUFs to run:                                                                                                                                                 |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Qwen3-**4B-2507**        | [Instruct](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF) • [Thinking](https://huggingface.co/unsloth/Qwen3-4B-Thinking-2507-GGUF)               |
| Qwen3-**30B-A3B**-2507   | [Instruct](#llama.cpp-run-qwen3-30b-a3b-instruct-2507-tutorial) • [Thinking](https://huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF)                 |
| Qwen3-**235B-A22B**-2507 | [Instruct](https://huggingface.co/unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF) • [Thinking](https://huggingface.co/unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF) |

## ⚙️Best Practices

{% hint style="success" %}
The settings for the Thinking and Instruct model are different.\
The thinking model uses temperature = 0.6, but the instruct model uses temperature = 0.7\
The thinking model uses top\_p = 0.95, but the instruct model uses top\_p = 0.8
{% endhint %}

To achieve optimal performance, Qwen recommends these settings:

| Instruct Model Settings:                                                                                      | Thinking Model Settings:                                                                                      |
| ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| <mark style="background-color:blue;">`Temperature = 0.7`</mark>                                               | <mark style="background-color:blue;">`Temperature = 0.6`</mark>                                               |
| `Min_P = 0.00` (llama.cpp's default is 0.1)                                                                   | `Min_P = 0.00` (llama.cpp's default is 0.1)                                                                   |
| `Top_P = 0.80`                                                                                                | `Top_P = 0.95`                                                                                                |
| `TopK = 20`                                                                                                   | `TopK = 20`                                                                                                   |
| `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) | `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) |

**Adequate Output Length**: Use an output length of `32,768` tokens for most queries, which is adequate for most queries.

Chat template for both Thinking (thinking has `<think></think>`) and Instruct is below:

```
<|im_start|>user
Hey there!<|im_end|>
<|im_start|>assistant
What is 1+1?<|im_end|>
<|im_start|>user
2<|im_end|>
<|im_start|>assistant
```

## 📖 Run Qwen3-30B-A3B-2507 Tutorials

Below are guides for the [Thinking](#thinking-qwen3-30b-a3b-thinking-2507) and [Instruct](#instruct-qwen3-30b-a3b-instruct-2507) versions of the model.

### Instruct: Qwen3-30B-A3B-Instruct-2507

Given that this is a non thinking model, there is no need to set `thinking=False` and the model does not generate `<think> </think>` blocks.

#### ⚙️Best Practices

To achieve optimal performance, Qwen recommends the following settings:

* We suggest using `temperature=0.7, top_p=0.8, top_k=20, and min_p=0.0` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions.
* **`temperature = 0.7`**
* `top_k = 20`
* `min_p = 0.00` (llama.cpp's default is 0.1)
* **`top_p = 0.80`**
* `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example.
* Supports up to `262,144` context natively but you can set it to `32,768` tokens for less RAM use

#### 🦙 Ollama: Run Qwen3-30B-A3B-Instruct-2507 Tutorial

1. Install `ollama` if you haven't already! You can only run models up to 32B in size.

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```bash
ollama run hf.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:UD-Q4_K_XL
```

#### :sparkles: Llama.cpp: Run Qwen3-30B-A3B-Instruct-2507 Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. You can directly pull from HuggingFace via:

   ```bash
   ./llama.cpp/llama-cli \
       -hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_XL \
       --jinja -ngl 99 --ctx-size 32768 \
       --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --presence-penalty 1.0
   ```
3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD\_Q4\_K\_XL or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF",
    local_dir = "unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

### Thinking: Qwen3-30B-A3B-Thinking-2507

This model supports only thinking mode and a 256K context window natively. The default chat template adds `<think>` automatically, so you may see only a closing `</think>` tag in the output.

#### ⚙️Best Practices

To achieve optimal performance, Qwen recommends the following settings:

* We suggest using `temperature=0.6, top_p=0.95, top_k=20, and min_p=0.0` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions.
* **`temperature = 0.6`**
* `top_k = 20`
* `min_p = 0.00` (llama.cpp's default is 0.1)
* **`top_p = 0.95`**
* `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example.
* Supports up to `262,144` context natively but you can set it to `32,768` tokens for less RAM use

#### 🦙 Ollama: Run Qwen3-30B-A3B-Instruct-2507 Tutorial

1. Install `ollama` if you haven't already! You can only run models up to 32B in size. To run the full 235B-A22B models, [see here](#run-qwen3-235b-a22b-instruct-2507).

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```bash
ollama run hf.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:UD-Q4_K_XL
```

#### :sparkles: Llama.cpp: Run Qwen3-30B-A3B-Instruct-2507 Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. You can directly pull from Hugging Face via:

   ```bash
   ./llama.cpp/llama-cli \
       -hf unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:Q4_K_XL \
       --jinja -ngl 99 --ctx-size 32768 \
       --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence-penalty 1.0
   ```
3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD\_Q4\_K\_XL or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF",
    local_dir = "unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

## 📖 Run **Qwen3-235B-A22B-2507** Tutorials

Below are guides for the [Thinking](#run-qwen3-235b-a22b-thinking-via-llama.cpp) and [Instruct](#run-qwen3-235b-a22b-instruct-via-llama.cpp) versions of the model.

### Thinking: Qwen3-**235B-A22B**-Thinking-2507

This model supports only thinking mode and a 256K context window natively. The default chat template adds `<think>` automatically, so you may see only a closing `</think>` tag in the output.

#### :gear: Best Practices

To achieve optimal performance, Qwen recommends these settings for the Thinking model:

* **`temperature = 0.6`**
* `top_k = 20`
* `min_p = 0.00` (llama.cpp's default is 0.1)
* `top_p = 0.95`
* `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example.
* **Adequate Output Length**: Use an output length of `32,768` tokens for most queries, which is adequate for most queries.

#### :sparkles:Run Qwen3-235B-A22B-Thinking via llama.cpp:

For Qwen3-235B-A22B, we will specifically use Llama.cpp for optimized inference and a plethora of options.

{% hint style="success" %}
If you want a **full precision unquantized version**, use our `Q8_K_XL, Q8_0` or `BF16` versions!
{% endhint %}

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

   ```bash
   apt-get update
   apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
   git clone https://github.com/ggml-org/llama.cpp
   cmake llama.cpp -B llama.cpp/build \
       -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
   cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
   cp llama.cpp/build/bin/llama-* llama.cpp
   ```
2. You can directly use llama.cpp to download the model but I normally suggest using `huggingface_hub` To use llama.cpp directly, do:

   ```bash
   ./llama.cpp/llama-cli \
       -hf unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF:Q2_K_XL \
       --ctx-size 16384 \
       --n-gpu-layers 99 \
       -ot ".ffn_.*_exps.=CPU" \
       --temp 0.6 \
       --min-p 0.0 \
       --top-p 0.95 \
       --top-k 20 \
       --presence-penalty 1.0
   ```
3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions..

   ```python
   # !pip install huggingface_hub hf_transfer
   import os
   os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
   from huggingface_hub import snapshot_download
   snapshot_download(
       repo_id = "unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF",
       local_dir = "unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF",
       allow_patterns = ["*UD-Q2_K_XL*"],
   )
   ```
4. Run the model and try any prompt.
5. Edit `--threads -1` for the number of CPU threads, `--ctx-size` 262114 for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% hint style="success" %}
Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.
{% endhint %}

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3-235B-A22B-Thinking-2507-GGUF/UD-Q2_K_XL/Qwen3-235B-A22B-Thinking-2507-UD-Q2_K_XL-00001-of-00002.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --seed 3407 \
    --temp 0.6 \
    --min-p 0.0 \
    --top-p 0.95 \
    --top-k 20
    --presence-penalty 1.0
```

{% endcode %}

### Instruct: Qwen3-**235B-A22B**-Instruct-2507

Given that this is a non thinking model, there is no need to set `thinking=False` and the model does not generate `<think> </think>` blocks.

#### ⚙️Best Practices

To achieve optimal performance, we recommend the following settings:

**1. Sampling Parameters**: We suggest using `temperature=0.7, top_p=0.8, top_k=20, and min_p=0.` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions.

2\. **Adequate Output Length**: We recommend using an output length of `16,384` tokens for most queries, which is adequate for instruct models.

3\. **Standardize Output Format:** We recommend using prompts to standardize model outputs when benchmarking.

* **Math Problems**: Include `Please reason step by step, and put your final answer within \boxed{}.` in the prompt.
* **Multiple-Choice Questions**: Add the following JSON structure to the prompt to standardize responses: "Please show your choice in the \`answer\` field with only the choice letter, e.g., \`"answer": "C".

#### :sparkles:Run Qwen3-235B-A22B-Instruct via llama.cpp:

For Qwen3-235B-A22B, we will specifically use Llama.cpp for optimized inference and a plethora of options.

{% hint style="info" %}
If you want a **full precision unquantized version**, use our `Q8_K_XL, Q8_0` or `BF16` versions!
{% endhint %}

1. Obtain the latest llama.cpp on [GitHub](https://github.com/ggml-org/llama.cpp) here. You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2\. You can directly use llama.cpp to download the model but I normally suggest using `huggingface_hub` To use llama.cpp directly, do:\\

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF:Q2_K_XL \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --temp 0.7 \
    --min-p 0.0 \
    --top-p 0.8 \
    --top-k 20 \
    --repeat-penalty 1.0
```

3\. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions..

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF",
    local_dir = "unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF",
    allow_patterns = ["*UD-Q2_K_XL*"],
)
```

4\. Run the model and try any prompt. 5. Edit `--threads -1` for the number of CPU threads, `--ctx-size` 262114 for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% hint style="success" %}
Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.
{% endhint %}

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF/UD-Q2_K_XL/Qwen3-235B-A22B-Instruct-2507-UD-Q2_K_XL-00001-of-00002.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --temp 0.7 \
    --min-p 0.0 \
    --top-p 0.8 \
    --top-k 20
```

{% endcode %}

### 🛠️ Improving generation speed <a href="#improving-generation-speed" id="improving-generation-speed"></a>

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.

The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel). You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster. The [next section](#how-to-fit-long-context-256k-to-1m) talks about KV cache quantization.

### 📐How to fit long context <a href="#how-to-fit-long-context-256k-to-1m" id="how-to-fit-long-context-256k-to-1m"></a>

To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below.

`--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`

You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1` So try out `--cache-type-k q4_1`

You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. After installing Flash Attention, you can then use `--cache-type-v q4_1`

## 🦥 Fine-tuning Qwen3-2507 with Unsloth

Unsloth makes [Qwen3](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth) and Qwen3-2507 fine-tuning 2x faster, use 70% less VRAM and supports 8x longer context lengths. Because Qwen3-2507 was only released in a 30B variant, this means you will need about a 40GB A100 GPU to fine-tune the model using QLoRA (4-bit).

For a notebook, because the model cannot fit in Colab's free 16GB GPUs, you will need to utilize a 40GB A100. You can utilize our Conversational notebook but replace the dataset to any of your using. This time you do not need to combined reasoning in your dataset as the model has no reasoning.

* [Qwen3 (14B) Reasoning + Conversational notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(14B\)-Reasoning-Conversational.ipynb)

If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth:

```bash
pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo
```

### Qwen3-2507 MOE models fine-tuning

Fine-tuning support includes MOE models: 30B-A3B and 235B-A22B. Qwen3-30B-A3B works on 30GB VRAM with Unsloth. On fine-tuning MoE's - it's probably not a good idea to fine-tune the router layer so we disabled it by default.

**Qwen3-2507-4B notebooks for:** [Thinking](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-Thinking.ipynb) and [Instruct](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(4B\)-Instruct.ipynb)

The 30B-A3B fits in 30GB VRAM, but you may lack RAM or disk space since the full 16-bit model must be downloaded and converted to 4-bit on the fly for QLoRA fine-tuning. This is due to issues importing 4-bit BnB MOE models directly. This only affects MOE models.

{% hint style="warning" %}
If you're fine-tuning the MOE models, please use `FastModel` and not `FastLanguageModel`
{% endhint %}

```python
from unsloth import FastModel
import torch
model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3-30B-A3B-Instruct-2507",
    max_seq_length = 2048, # Choose any for long context!
    load_in_4bit = True,  # 4 bit quantization to reduce memory
    load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory
    full_finetuning = False, # [NEW!] We have full finetuning now!
    # token = "hf_...", # use one if using gated models
)
```

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-be34c101c627020c7a6cfb6cd249f2462587d235%2Fqwen33%20mascot.png?alt=media" alt=""><figcaption></figcaption></figure>


# MiniMax-M2.7 - How to Run Locally

Run MiniMax-M2.7 LLM locally on your own device!

MiniMax-M2.7 is a new open model for agentic coding and chat use-cases. The model achieves SOTA performance in SWE-Pro (56.22%) and Terminal Bench 2 (57.0%).

The **230B parameters** (10B active) model is the successor to [MiniMax-M25](/docs/models/tutorials/minimax-m25) and has a **200K context** window. The unquantized bf16 requires **457GB**. Unsloth Dynamic **4-bit** GGUF reduces the size to **108GB** **(-60%)** so it can run on a **128GB RAM** devic&#x65;**:** [**MiniMax-M2.7 GGUF**](https://huggingface.co/unsloth/MiniMax-M2.7-GGUF)

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA quantization performance - so important layers are upcasted to higher bits (e.g. 8 or 16-bit). Thank you MiniMax for day zero access.

{% hint style="success" %}
NEW MiniMax-M2.7 GGUF Benchmarks available! [See here](#gguf-benchmarks)
{% endhint %}

### :gear: Usage Guide

The 4-bit dynamic quant `UD-IQ4_XS` uses **108GB** of disk space - this fits nicely on a **128GB unified memory Mac** for \~15+ tokens/s, and also works faster with a **1x16GB GPU and 96GB of RAM** for 25+ tokens/s. **2-bit** quants or the biggest 2-bit will fit on a 96GB device.

For near **full precision**, use `Q8_0` (8-bit) which utilizes 243GB and will fit on a 256GB RAM device / Mac for 15+ tokens/s.

{% hint style="success" %}
For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower.
{% endhint %}

### Recommended Settings

MiniMax recommends using the following parameters for best performance: `temperature=1.0`, `top_p = 0.95`, `top_k = 40`.

{% columns %}
{% column %}

| Default Settings (Most Tasks) |
| ----------------------------- |
| `temperature = 1.0`           |
| `top_p = 0.95`                |
| `top_k = 40`                  |
| {% endcolumn %}               |

{% column %}

* **Maximum context window:** `196,608`
* Default system prompt:

{% code overflow="wrap" %}

```
You are a helpful assistant. Your name is MiniMax-M2.7 and is built by MiniMax.
```

{% endcode %}
{% endcolumn %}
{% endcolumns %}

## Run MiniMax-M2.7 Tutorials:

To make MiniMax-M2.7 work on a 128GB RAM device, we will be utilizing the 4-bit [`UD-IQ4_XS` quant](https://huggingface.co/unsloth/MiniMax-M2.7-GGUF?show_file_info=UD-IQ4_XS%2FMiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf). You can now run MiniMax-M2.7 in [llama.cpp](#run-in-llama.cpp) and [Unsloth Studio](#run-in-unsloth-studio).

{% hint style="warning" %}
Do NOT use CUDA 13.2 to run any model as it may cause gibberish or poor outputs. NVIDIA is working on a fix.
{% endhint %}

### 🦥 Run in Unsloth Studio

MiniMax-M2.7 can now run in [Unsloth Studio](/docs/new/studio), our new open-source web UI for local AI. Unsloth Studio lets you run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio#run-models-locally) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Uses llama.cpp for Fast CPU + GPU inference and CPU offloading
  {% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FstfdTMsoBMmsbQsgQ1Ma%2Flandscape%20clip%20gemma.gif?alt=media&amp;token=eec5f2f7-b97a-4c1c-ad01-5a041c3e4013" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

Run in your terminal:

**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows PowerShell:**

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Launch Unsloth

**MacOS, Linux, WSL and Windows:**

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

**Then open `http://localhost:8888` in your browser.**
{% endstep %}

{% step %}

#### Search and download MiniMax-M2.7

On first launch you will need to create a password to secure your account and sign in again later. You’ll then see a brief onboarding wizard to choose a model, dataset, and basic settings. You can skip it at any time.

You can choose `UD-IQ4_XS` (dynamic 4bit quant) or other quantized versions like `UD-Q4_K_XL` . If downloads get stuck, see [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

Then go to the [Unsloth Chat](/docs/new/studio/chat) tab and search for MiniMax-M2.7 in the search bar and download your desired model and quant. It will take some time to download due to the size so please wait. To ensure fast inference, ensure you have [enough RAM/VRAM](#usage-guide), otherwise inference will still work, but Unsloth will offload to your CPU.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fh6qv7Mh2VqtdhZaixrnO%2FScreenshot%202026-04-11%20at%206.46.55%E2%80%AFPM.png?alt=media&amp;token=e2568c00-86eb-452f-a4eb-10bcc0194ddf" alt=""><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

#### Run MiniMax-M2.7

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth Studio inference guide](/docs/new/studio/chat).
{% endstep %}
{% endstepper %}

### ✨ Run in llama.cpp

{% hint style="warning" %}
Do NOT use CUDA 13.2 to run any model as it may cause gibberish or poor outputs. NVIDIA is working on a fix.
{% endhint %}

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:IQ4\_XS) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 200K context length.

Follow this for **most default** use-cases:

```bash
export LLAMA_CACHE="unsloth/MiniMax-M2.7-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/MiniMax-M2.7-GGUF:UD-IQ4_XS \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 40
```

{% endstep %}

{% step %}
Download the model (after installing `pip install huggingface_hub hf_transfer`). You can choose UD-IQ4\_XS (dynamic 4-bit quant) or other quantized versions like `UD-Q6_K_XL` . We recommend using our 4bit dynamic quant UD-IQ4\_XS to balance size and accuracy. If downloads get stuck, see [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

```bash
hf download unsloth/MiniMax-M2.7-GGUF \
    --local-dir unsloth/MiniMax-M2.7-GGUF \
    --include "*UD-IQ4_XS*" # Use "*Q8_0*" for 8-bit
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/MiniMax-M2.7-GGUF/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 40
```

{% endcode %}
{% endstep %}
{% endstepper %}

#### 🦙 Llama-server & OpenAI's completion library

To deploy MiniMax-M2.7 for production, we use `llama-server` or OpenAI API. In a new terminal say via tmux, deploy the model via:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/MiniMax-M2.7-GGUF/UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf \
    --alias "unsloth/MiniMax-M2.7" \
    --prio 3 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40 \
    --port 8001
```

{% endcode %}

Then in a new terminal, after doing `pip install openai`, do:

{% code overflow="wrap" %}

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/MiniMax-M2.7",
    messages = [{"role": "user", "content": "Create a Snake game."},],
)
print(completion.choices[0].message.content)
```

{% endcode %}

## 📊 Benchmarks

### GGUF Benchmarks

Below are KLD 99% benchmarks for MiniMax-M2.7. Lower left is better:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FOkxq866NI68lXd0LBcsV%2FCode_Generated_Image(18)(1).png?alt=media&amp;token=ce73524f-1db2-410a-ac2f-a26cba04d270" alt=""><figcaption></figcaption></figure>

Because MiniMax-M2.7 utilizes the same architecture as MiniMax-M2.5, GGUF quantization benchmarks for M2.7 should be very similar to M2.5. So, we'll also refer to previous quant benchmark conducted for M2.5:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhfO2gsbz2lWrZXg3ojyE%2FHCGBTzgboAASv_A.png?alt=media&amp;token=7d6334ca-4f3c-4946-aacd-d55527375fce" alt=""><figcaption></figcaption></figure>

[Benjamin Marie (third-party) benchmarked](https://x.com/bnjmn_marie/status/2027043753484021810/photo/1) **MiniMax-M2.5** using **Unsloth GGUF quantizations** on a **750-prompt mixed suite** (LiveCodeBench v6, MMLU Pro, GPQA, Math500), reporting both **overall accuracy** and **relative error increase** (how much more often the quantized model makes mistakes vs. the original).

Unsloth quants, no matter their precision perform much better than their non-Unsloth counterparts for both accuracy and relative error (despite being 8GB smaller).

**Key results:**

* **Best quality/size tradeoff here: `unsloth UD-Q4_K_XL`.**\
  It’s the closest to Original: only **6.0 points** down, and “only” **+22.8%** more errors than baseline.
* **Other Unsloth Q4 quants perform closely together (\~64.5–64.9 accuracy).**\
  `IQ4_NL`, `MXFP4_MOE`, and `UD-IQ2_XXS` are all basically the same quality on this benchmark, with **\~33–35%** more errors than Original.
* Unsloth GGUFs perform much better than other non-Unsloth GGUFs, e.g. see `lmstudio-community - Q4_K_M` (despite being 8GB smaller) and `AesSedai - IQ3_S`.

### Official Benchmarks

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fn5Xz2P6kzHRH2sQGPsHH%2Fminimaxm2.7%20model.jpg?alt=media&amp;token=04f4b3fd-9d04-4e80-9f06-09afd8ce884d" alt=""><figcaption></figcaption></figure>


# GLM-5: How to Run Locally Guide

Run the new GLM-5 model by Z.ai on your own local device!

GLM-5 is Z.ai’s latest reasoning model, delivering stronger coding, agent, and chat performance than [GLM-4.7](/docs/models/tutorials/glm-4.7), and is designed for long context reasoning. It increases performance on benchmarks such as Humanity's Last Exam 50.4% (+7.6%), BrowseComp 75.9% (+8.4%) and Terminal-Bench-2.0 61.1% (+28.3%).

The full 744B parameter (40B active) model has a **200K context** window and was pre-trained on 28.5T tokens. The full GLM-5 model requires **1.65TB** of disk space, while the Unsloth Dynamic 2-bit GGUF reduces the size to **241GB** **(-85%)**, and dynamic **1-bit is 176GB (-89%):** [**GLM-5-GGUF**](https://huggingface.co/unsloth/GLM-5-GGUF)

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA quantization performance - so 1-bit has important layers upcasted to 8 or 16-bit. Thank you Z.ai for providing Unsloth with day zero access.

### :gear: Usage Guide

The 2-bit dynamic quant UD-IQ2\_XXS uses **241GB** of disk space - this can directly fit on a **256GB unified memory Mac**, and also works well in a **1x24GB card and 256GB of RAM** with MoE offloading. The **1-bit** quant will fit on a 180GB RAM and 8-bit requires 805GB RAM.

{% hint style="success" %}
For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower.
{% endhint %}

### Recommended Settings

Use distinct settings for different use cases:

| Default Settings (Most Tasks)    | SWE Bench Verified               |
| -------------------------------- | -------------------------------- |
| temperature = 1.0                | temperature = 0.7                |
| top\_p = 0.95                    | top\_p = 1.0                     |
| max new tokens = 131072          | max new tokens = 16384           |
| repeat penalty = disabled or 1.0 | repeat penalty = disabled or 1.0 |

* `Min_P = 0.01` (llama.cpp's default is 0.05)
* **Maximum context window:** `202,752`.
* For multi-turn agentic tasks (τ²-Bench and Terminal Bench 2), please turn on Preserved\
  Thinking mode.

## Run GLM-5 Tutorials:

#### ✨ Run in llama.cpp

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` **on** [**GitHub here**](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:IQ2\_XXS) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 200K context length.

Follow this for **general instruction** use-cases:

```bash
export LLAMA_CACHE="unsloth/GLM-5-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/GLM-5-GGUF:UD-IQ2_XXS \
    --ctx-size 16384 \
    --flash-attn on \
    --temp 0.7 \
    --top-p 1.0 \
    --min-p 0.01
```

Follow this for **tool-calling** use-cases:

```bash
export LLAMA_CACHE="unsloth/GLM-5-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/GLM-5-GGUF:UD-IQ2_XXS \
    --ctx-size 16384 \
    --flash-attn on \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-Q2_K_XL` (dynamic 2bit quant) or other quantized versions like `UD-Q4_K_XL` . We <mark style="background-color:green;">**recommend using our 2bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>. If downloads get stuck, see [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

```bash
pip install -U huggingface_hub
hf download unsloth/GLM-5-GGUF \
    --local-dir unsloth/GLM-5-GGUF \
    --include "*UD-IQ2_XXS*" # Use "*UD-TQ1_0*" for Dynamic 1bit
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/GLM-5-GGUF/UD-IQ2_XXS/GLM-5-UD-IQ2_XXS-00001-of-00006.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407
```

{% endcode %}
{% endstep %}
{% endstepper %}

### 🦙 Llama-server serving & OpenAI's completion library

To deploy GLM-5 for production, we use `llama-server` In a new terminal say via tmux, deploy the model via:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/GLM-5-GGUF/UD-IQ2_XXS/GLM-5-UD-IQ2_XXS-00001-of-00006.gguf \
    --alias "unsloth/GLM-5" \
    --prio 3 \
    --temp 1.0 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --port 8001
```

{% endcode %}

Then in a new terminal, after doing `pip install openai`, do:

{% code overflow="wrap" %}

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/GLM-5",
    messages = [{"role": "user", "content": "Create a Snake game."},],
)
print(completion.choices[0].message.content)
```

{% endcode %}

And you will get the following example of a Snake game:

{% columns %}
{% column width="58.333333333333336%" %}
{% code expandable="true" %}

````markdown
Here is a complete, playable Snake game contained within a single HTML file. You can copy this code, save it as an `.html` file (e.g., `snake.html`), and open it in your web browser to play.

### The Code

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Classic Snake Game</title>
    <style>
        body {
            display: flex;
            flex-direction: column;
            justify-content: center;
            align-items: center;
            height: 100vh;
            margin: 0;
            background-color: #222;
            color: white;
            font-family: Arial, sans-serif;
        }

        #gameCanvas {
            border: 2px solid #fff;
            background-color: #000;
        }

        h1 {
            margin-bottom: 10px;
        }

        #scoreBoard {
            font-size: 20px;
            margin-bottom: 10px;
        }

        #gameOverMenu {
            position: absolute;
            display: none;
            flex-direction: column;
            justify-content: center;
            align-items: center;
            background: rgba(0, 0, 0, 0.85);
            padding: 20px;
            border-radius: 10px;
            border: 2px solid red;
        }

        button {
            margin-top: 15px;
            padding: 10px 20px;
            font-size: 16px;
            cursor: pointer;
            background-color: #4CAF50;
            color: white;
            border: none;
            border-radius: 5px;
        }
        
        button:hover {
            background-color: #45a049;
        }
    </style>
</head>
<body>

    <h1>Snake Game</h1>
    <div id="scoreBoard">Score: 0</div>
    <canvas id="gameCanvas" width="400" height="400"></canvas>

    <div id="gameOverMenu">
        <h2 style="color: red; margin: 0;">Game Over!</h2>
        <p id="finalScore">Final Score: 0</p>
        <button onclick="resetGame()">Play Again</button>
    </div>

    <script>
        // Game Constants
        const canvas = document.getElementById('gameCanvas');
        const ctx = canvas.getContext('2d');
        const scoreBoard = document.getElementById('scoreBoard');
        const gameOverMenu = document.getElementById('gameOverMenu');
        const finalScoreDisplay = document.getElementById('finalScore');

        const gridSize = 20; // Size of each square
        const tileCount = canvas.width / gridSize; // Number of squares per row/column

        // Game Variables
        let dx = 0; // Horizontal velocity
        let dy = 0; // Vertical velocity
        let score = 0;
        let snake = [];
        let foodX, foodY;
        let gameInterval;
        let isGameRunning = false;

        // Initialize the game
        function initGame() {
            snake = [
                {x: 10, y: 10}, 
                {x: 9, y: 10}, 
                {x: 8, y: 10}
            ];
            score = 0;
            scoreBoard.innerText = 'Score: ' + score;
            dx = 1; // Start moving right immediately
            dy = 0;
            placeFood();
            isGameRunning = true;
            gameOverMenu.style.display = 'none';
            
            // Start the game loop
            if (gameInterval) clearInterval(gameInterval);
            gameInterval = setInterval(gameLoop, 100); // Run game loop every 100ms
        }

        // Main game loop
        function gameLoop() {
            if (!isGameRunning) return;

            moveSnake();
            if (checkGameOver()) {
                endGame();
                return;
            }
            checkFoodCollision();
            draw();
        }

        // Move the snake
        function moveSnake() {
            // Create new head based on current direction
            const head = {x: snake[0].x + dx, y: snake[0].y + dy};
            
            // Add new head to the beginning of the array
            snake.unshift(head);

            // Remove the tail (last element) unless food is eaten
            // Note: We handle removing the tail in checkFoodCollision
            snake.pop(); 
        }

        // Check if snake eats food
        function checkFoodCollision() {
            const head = snake[0];
            
            if (head.x === foodX && head.y === foodY) {
                // Grow snake: add a tail piece (duplicate the last one)
                snake.push({...snake[snake.length - 1]});
                score += 10;
                scoreBoard.innerText = 'Score: ' + score;
                placeFood();
            }
        }

        // Check for collisions (walls or self)
        function checkGameOver() {
            const head = snake[0];

            // Wall collision
            if (head.x < 0 || head.x >= tileCount || head.y < 0 || head.y >= tileCount) {
                return true;
            }

            // Self collision (start checking from the 4th segment because head can't hit the first 3)
            for (let i = 4; i < snake.length; i++) {
                if (head.x === snake[i].x && head.y === snake[i].y) {
                    return true;
                }
            }

            return false;
        }

        // Draw everything
        function draw() {
            // Clear canvas
            ctx.fillStyle = 'black';
            ctx.fillRect(0, 0, canvas.width, canvas.height);

            // Draw Food
            ctx.fillStyle = 'red';
            ctx.fillRect(foodX * gridSize, foodY * gridSize, gridSize - 2, gridSize - 2);

            // Draw Snake
            ctx.fillStyle = 'lime';
            for (let i = 0; i < snake.length; i++) {
                // Draw the head slightly different or just standard
                const part = snake[i];
                ctx.fillRect(part.x * gridSize, part.y * gridSize, gridSize - 2, gridSize - 2);
            }
        }

        // Place food at random position
        function placeFood() {
            foodX = Math.floor(Math.random() * tileCount);
            foodY = Math.floor(Math.random() * tileCount);

            // Ensure food doesn't spawn on the snake body
            for (let part of snake) {
                if (part.x === foodX && part.y === foodY) {
                    placeFood(); // Recursively find a new spot
                    return;
                }
            }
        }

        // End game logic
        function endGame() {
            isGameRunning = false;
            clearInterval(gameInterval);
            finalScoreDisplay.innerText = 'Final Score: ' + score;
            gameOverMenu.style.display = 'flex';
        }

        // Reset game logic
        function resetGame() {
            initGame();
        }

        // Keyboard controls
        document.addEventListener('keydown', (e) => {
            // Prevent reversing direction (can't go left if going right)
            switch(e.key) {
                case 'ArrowUp':
                    if (dy !== 1) { dx = 0; dy = -1; }
                    break;
                case 'ArrowDown':
                    if (dy !== -1) { dx = 0; dy = 1; }
                    break;
                case 'ArrowLeft':
                    if (dx !== 1) { dx = -1; dy = 0; }
                    break;
                case 'ArrowRight':
                    if (dx !== -1) { dx = 1; dy = 0; }
                    break;
                case ' ':
                    if (!isGameRunning && gameOverMenu.style.display !== 'flex') {
                        initGame();
                    }
                    break;
            }
        });

        // Start the game on load
        initGame();
    </script>
</body>
</html>
```

### How to Play
1.  **Copy the code** above.
2.  Create a new file on your computer named `snake.html`.
3.  **Paste the code** into that file and save it.
4.  **Double-click `snake.html`** to open it in your browser.

### Controls
*   **Arrow Keys**: Move Up, Down, Left, Right.
*   **Spacebar**: Starts the game (if it hasn't started yet).
*   **Play Again Button**: Appears when you crash to restart the game.

### Features of this Version
*   **Grid-based movement**: Classic retro feel.
*   **Score tracking**: Updates in real-time.
*   **Game Over Screen**: Displays your final score and allows you to restart easily.
*   **Collision Detection**: Ends the game if you hit the walls or yourself.
*   **Self-Collision Safety**: The code prevents the snake from accidentally eating itself immediately after eating food due to "tail skipping" logic commonly found in simple tutorials.
````

{% endcode %}
{% endcolumn %}

{% column width="41.666666666666664%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsdUp9wbpqDx0Lhp00xZ0%2Fimage.png?alt=media&amp;token=a5e67ac2-65bf-43e0-8c13-4aef9a7d269e" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### :computer: vLLM Deployment

You can now serve Z.ai's FP8 version of the model via vLLM. You need 860GB VRAM or more, so 8xH200 (141x8 = 1128GB) is at least recommended. 8xB200 works well. Firstly, install vllm nightly:

{% code overflow="wrap" %}

```bash
uv pip install --upgrade --force-reinstall vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly/cu130
uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git
uv pip install --force-reinstall numba
```

{% endcode %}

To disable FP8 KV Cache (reduces memory usage by 50%), remove `--kv-cache-dtype fp8`

```bash
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
vllm serve unsloth/GLM-5-FP8 \
    --served-model-name unsloth/GLM-5-FP8 \
    --kv-cache-dtype fp8 \
    --tensor-parallel-size 8 \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice \
    --dtype bfloat16 \
    --seed 3407 \
    --max-model-len 200000 \
    --gpu-memory-utilization 0.93 \
    --max_num_batched_tokens 4096 \
    --speculative-config.method mtp \
    --speculative-config.num_speculative_tokens 1 \
    --port 8001
```

You can then call the served model via the OpenAI API:

```python
from openai import AsyncOpenAI, OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8001/v1"
client = OpenAI( # or AsyncOpenAI
    api_key = openai_api_key,
    base_url = openai_api_base,
)
```

### :hammer:Tool Calling with GLM 5

See [Tool Calling Guide](/docs/basics/tool-calling-guide-for-local-llms) for more details on how to do tool calling. In a new terminal (if using tmux, use CTRL+B+D), we create some tools like adding 2 numbers, executing Python code, executing Linux functions and much more:

{% code expandable="true" %}

```python
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def subtract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([
        "A long time ago in a galaxy far far away...",
        "There were 2 friends who loved sloths and code...",
        "The world was ending because every sloth evolved to have superhuman intelligence...",
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "subtract_number": subtract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [
    {
        "type": "function",
        "function": {
            "name": "add_number",
            "description": "Add two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "multiply_number",
            "description": "Multiply two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "subtract_number",
            "description": "Subtract two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "write_a_story",
            "description": "Writes a random story.",
            "parameters": {
                "type": "object",
                "properties": {},
                "required": [],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "terminal",
            "description": "Perform operations from the terminal.",
            "parameters": {
                "type": "object",
                "properties": {
                    "command": {
                        "type": "string",
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",
                    },
                },
                "required": ["command"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "python",
            "description": "Call a Python interpreter with some Python code that will be ran.",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {
                        "type": "string",
                        "description": "The Python code to run",
                    },
                },
                "required": ["code"],
            },
        },
    },
]
```

{% endcode %}

We then use the below functions (copy and paste and execute) which will parse the function calls automatically and call the OpenAI endpoint for any model:

{% code overflow="wrap" expandable="true" %}

```python
from openai import OpenAI
def unsloth_inference(
    messages,
    temperature = 1.0,
    top_p = 0.95,
    top_k = -1,
    min_p = 0.01,
    repetition_penalty = 1.0,
):
    messages = messages.copy()
    openai_client = OpenAI(
        base_url = "http://127.0.0.1:8001/v1",
        api_key = "sk-no-key-required",
    )
    model_name = next(iter(openai_client.models.list())).id
    print(f"Using model = {model_name}")
    has_tool_calls = True
    original_messages_len = len(messages)
    while has_tool_calls:
        print(f"Current messages = {messages}")
        response = openai_client.chat.completions.create(
            model = model_name,
            messages = messages,
            temperature = temperature,
            top_p = top_p,
            tools = tools if tools else None,
            tool_choice = "auto" if tools else None,
            extra_body = {"top_k": top_k, "min_p": min_p, "repetition_penalty" :repetition_penalty,}
        )
        tool_calls = response.choices[0].message.tool_calls or []
        content = response.choices[0].message.content or ""
        tool_calls_dict = [tc.to_dict() for tc in tool_calls] if tool_calls else tool_calls
        messages.append({"role": "assistant", "tool_calls": tool_calls_dict, "content": content,})
        for tool_call in tool_calls:
            fx, args, _id = tool_call.function.name, tool_call.function.arguments, tool_call.id
            out = MAP_FN[fx](**json.loads(args))
            messages.append({"role": "tool", "tool_call_id": _id, "name": fx, "content": str(out),})
        else:
            has_tool_calls = False
    return messages
```

{% endcode %}

After launching GLM 5 via `llama-server` like in [#deploy-with-llama-server-and-openais-completion-library](#deploy-with-llama-server-and-openais-completion-library "mention") or see [Tool Calling Guide](/docs/basics/tool-calling-guide-for-local-llms) for more details, we then can do some tool calls.

### 📊 Benchmarks

You can view further below for benchmarks in table format:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F3QI8akFZoQFXsZ2ojtgK%2Fglm5%20bench.jpg?alt=media&amp;token=0fb5d73f-4dc4-46f5-bd76-206c26ff5e96" alt="" width="375"><figcaption></figcaption></figure>

<table data-full-width="true"><thead><tr><th>Benchmark</th><th>GLM-5</th><th>GLM-4.7</th><th>DeepSeek-V3.2</th><th>Kimi K2.5</th><th>Claude Opus 4.5</th><th>Gemini 3 Pro</th><th>GPT-5.2 (xhigh)</th></tr></thead><tbody><tr><td>HLE</td><td>30.5</td><td>24.8</td><td>25.1</td><td>31.5</td><td>28.4</td><td>37.2</td><td>35.4</td></tr><tr><td>HLE (w/ Tools)</td><td>50.4</td><td>42.8</td><td>40.8</td><td>51.8</td><td>43.4*</td><td>45.8*</td><td>45.5*</td></tr><tr><td>AIME 2026 I</td><td>92.7</td><td>92.9</td><td>92.7</td><td>92.5</td><td>93.3</td><td>90.6</td><td>-</td></tr><tr><td>HMMT Nov. 2025</td><td>96.9</td><td>93.5</td><td>90.2</td><td>91.1</td><td>91.7</td><td>93.0</td><td>97.1</td></tr><tr><td>IMOAnswerBench</td><td>82.5</td><td>82.0</td><td>78.3</td><td>81.8</td><td>78.5</td><td>83.3</td><td>86.3</td></tr><tr><td>GPQA-Diamond</td><td>86.0</td><td>85.7</td><td>82.4</td><td>87.6</td><td>87.0</td><td>91.9</td><td>92.4</td></tr><tr><td>SWE-bench Verified</td><td>77.8</td><td>73.8</td><td>73.1</td><td>76.8</td><td>80.9</td><td>76.2</td><td>80.0</td></tr><tr><td>SWE-bench Multilingual</td><td>73.3</td><td>66.7</td><td>70.2</td><td>73.0</td><td>77.5</td><td>65.0</td><td>72.0</td></tr><tr><td>Terminal-Bench 2.0 (Terminus 2)</td><td>56.2 / 60.7 †</td><td>41.0</td><td>39.3</td><td>50.8</td><td>59.3</td><td>54.2</td><td>54.0</td></tr><tr><td>Terminal-Bench 2.0 (Claude Code)</td><td>56.2 / 61.1 †</td><td>32.8</td><td>46.4</td><td>-</td><td>57.9</td><td>-</td><td>-</td></tr><tr><td>CyberGym</td><td>43.2</td><td>23.5</td><td>17.3</td><td>41.3</td><td>50.6</td><td>39.9</td><td>-</td></tr><tr><td>BrowseComp</td><td>62.0</td><td>52.0</td><td>51.4</td><td>60.6</td><td>37.0</td><td>37.8</td><td>-</td></tr><tr><td>BrowseComp (w/ Context Manage)</td><td>75.9</td><td>67.5</td><td>67.6</td><td>74.9</td><td>67.8</td><td>59.2</td><td>65.8</td></tr><tr><td>BrowseComp-Zh</td><td>72.7</td><td>66.6</td><td>65.0</td><td>62.3</td><td>62.4</td><td>66.8</td><td>76.1</td></tr><tr><td>τ²-Bench</td><td>89.7</td><td>87.4</td><td>85.3</td><td>80.2</td><td>91.6</td><td>90.7</td><td>85.5</td></tr><tr><td>MCP-Atlas (Public Set)</td><td>67.8</td><td>52.0</td><td>62.2</td><td>63.8</td><td>65.2</td><td>66.6</td><td>68.0</td></tr><tr><td>Tool-Decathlon</td><td>38.0</td><td>23.8</td><td>35.2</td><td>27.8</td><td>43.5</td><td>36.4</td><td>46.3</td></tr><tr><td>Vending Bench 2</td><td>$4,432.12</td><td>$2,376.82</td><td>$1,034.00</td><td>$1,198.46</td><td>$4,967.06</td><td>$5,478.16</td><td>$3,591.33</td></tr></tbody></table>


# Kimi K2.5: How to Run Locally Guide

Guide on running Kimi-K2.5 on your own local device!

Kimi-K2.5 is the new model by Moonshot which achieves SOTA performance in vision, coding, agentic and chat tasks. The 1T parameter hybrid reasoning model requires 600GB of disk space, while the quantized **Unsloth Dynamic 1.8-bit** version reduces this to 240GB (-60% size)**:** [**Kimi-K2.5-GGUF**](https://huggingface.co/unsloth/Kimi-K2.5-GGUF)

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA Aider and 5-shot MMLU performance. See how our Dynamic 1–2 bit GGUFs perform on [coding benchmarks](/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot).

### :gear: Recommended Requirements

{% hint style="info" %}
You need >**240GB of disk space** to run the 1-bit quant!

For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower.
{% endhint %}

The 1.8-bit (UD-TQ1\_0) quant will run on a single 24GB GPU if you offload all MoE layers to system RAM (or a fast SSD). With \~256GB RAM, expect \~10 tokens/s. The full Kimi K2.5 model is 630GB and typically requires at least 4× H200 GPUs.

If the model fits, you will get >40 tokens/s when using a B200.

To run the model in near **full precision**, you can use the 4-bit or 5-bit quants. You can use any higher just to be safe.

For strong performance, aim for >240GB of unified memory (or combined RAM+VRAM) to reach 10+ tokens/s. If you’re below that, it'll work but speed will drop (llama.cpp can still run via mmap/disk offload) and may fall from \~10 tokens/s to <2 token/s.

We recommend UD-Q2\_K\_XL (375GB) as a good size/quality balance. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just slower due to offloading.

## 🥝 Run Kimi K2.5 Guide

Kimi-K2.5 requires different sampling parameters for different use-cases.

Currently there is **no vision support** for the model but hopefully llama.cpp will support it soon.

{% hint style="success" %}
**To run the model in full precision, you only need to use the 4-bit or 5-bit Dynamic GGUFs (e.g. UD\_Q4\_K\_XL) because the model was originally released in INT4 format.**

You can choose a higher-bit quantization just to be safe in case of small quantization differences, but in most cases this is unnecessary.
{% endhint %}

### 🌙 Usage Guide:

According to Moonshot AI, these are the recommended settings for Kimi K2.5 inference:

| Default Settings (Instant Mode)                                    | Thinking Mode                                                      |
| ------------------------------------------------------------------ | ------------------------------------------------------------------ |
| <mark style="background-color:green;">**temperature = 0.6**</mark> | <mark style="background-color:green;">**temperature = 1.0**</mark> |
| <mark style="background-color:green;">**top\_p = 0.95**</mark>     | <mark style="background-color:green;">**top\_p = 0.95**</mark>     |
| min\_p = 0.01                                                      | min\_p = 0.01                                                      |

* Set the **temperature 1.0** to reduce repetition and incoherence.
* Suggested context length = 98,304 (up to 256K)
* Note: Using different tools may require different settings

{% hint style="info" %}
We recommend setting <mark style="background-color:green;">**min\_p to 0.01**</mark> to suppress the occurrence of unlikely tokens with low probabilities. And **disable or set repeat penalty = 1.0** if needed.
{% endhint %}

#### Chat Template for Kimi K2.5

Running `tokenizer.apply_chat_template([{"role": "user", "content": "What is 1+1?"},])` gets:

{% code overflow="wrap" %}

```
<|im_system|>system<|im_middle|>You are Kimi, an AI assistant created by Moonshot AI.<|im_end|><|im_user|>user<|im_middle|>What is 1+1?<|im_end|><|im_assistant|>assistant<|im_middle|><think>
```

{% endcode %}

#### 🦥 Run Kimi-K2.5 in Unsloth Studio

Kimi-K2.5 can be run in [Unsloth Studio](/docs/new/studio), our new open-source web UI for local AI. With Unsloth Studio, you can run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio#run-models-locally) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFeQ0UUlnjXkDdqhcWglh%2Fskinny%20studio%20chat.png?alt=media&amp;token=c2ee045f-c243-4024-a8e4-bb4dbe7bae79" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}
**Install Unsloth**

Run in your terminal:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% hint style="success" %}
**Installation will be quick and take approx 1-2 mins.**
{% endhint %}
{% endstep %}

{% step %}
**Launch Unsloth**

MacOS, Linux, WSL and Windows:

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

Then open `http://localhost:8888` in your browser.
{% endstep %}

{% step %}
**Search and download Kimi-K2.5**

On first launch you will need to create a password to secure your account and sign in again later. You’ll then see a brief onboarding wizard to choose a model, dataset, and basic settings. You can skip it at any time and go directly to chat.

Then go to the [Unsloth Chat](/docs/new/studio/chat) tab and search for **Kimi-K2.5** in the search bar and download your desired model and quant. Ensure you have enough compute the run the model.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FexhsJAWYrkO20yACHDWR%2FScreenshot%202026-04-02%20at%202.05.22%E2%80%AFAM.png?alt=media&amp;token=387d1dc3-0c97-44c8-b8de-67fe90d9ade7" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}
**Run Kimi-K2.5**

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth Studio inference guide](/docs/new/studio/chat).

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FXPQGEEr1YoKofrTatAKK%2Ftoolcallingif.gif?alt=media&amp;token=25d68698-fb13-4c46-99b2-d39fb025df08" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}
{% endstepper %}

### ✨ Run Kimi K2.5 in llama.cpp

For this guide we'll be running the smallest 1-bit quant which is 240GB in size. Feel free to change quantization type to 2-bit, 3-bit etc. To run the model in near **full precision**, you can use the 4-bit or 5-bit quants. You can use any higher just to be safe.

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:UD-TQ1\_0) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location.

{% hint style="success" %}
`LLAMA_SET_ROWS=1` makes llama.cpp a little bit faster! Use it! `--fit on` auto fits models on all your GPUs and CPUs optimally.
{% endhint %}

```bash
export LLAMA_CACHE="unsloth/Kimi-K2.5-GGUF"
LLAMA_SET_ROWS=1 ./llama.cpp/llama-cli \
    -hf unsloth/Kimi-K2.5-GGUF:UD-TQ1_0\
    --temp 1.0 \
    --min-p 0.01 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --seed 3407
```

3. `--fit on` will auto fit the model to your system. If not using `--fit on` and you have around 360GB of combined GPU memory, remove `-ot ".ffn_.*_exps.=CPU"` to get maximum speed.

{% hint style="info" %}
Use `--fit on` for auto fitting on GPUs and CPUs. If this doesn't work, then see below:

Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). We recommend using our 2bit dynamic quant UD-Q2\_K\_XL to balance size and accuracy. All versions at: [huggingface.co/unsloth/Kimi-K2.5-GGUF](https://huggingface.co/unsloth/Kimi-K2.5-GGUF) If downloads get stuck, see [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

{% code overflow="wrap" %}

```bash
pip install -U huggingface_hub
hf download unsloth/Kimi-K2.5-GGUF \
    --local-dir unsloth/Kimi-K2.5-GGUF \
    --include "*UD-TQ1_0*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

{% endcode %}

{% hint style="info" %}
If you find that downloads get stuck at 90 to 95% or so, please see our [troubleshooting guide](https://docs.unsloth.ai/basics/troubleshooting-and-faqs#downloading-gets-stuck-at-90-to-95).
{% endhint %}

4. Run any prompt.
5. Edit  `--ctx-size 16384` for context length. You can also leave this out for auto context length discovery via `--fit on`

{% code overflow="wrap" %}

```bash
LLAMA_SET_ROWS=1 ./llama.cpp/llama-cli \
    --model unsloth/Kimi-K2.5-GGUF/UD-TQ1_0/Kimi-K2.5-UD-TQ1_0-00001-of-00005.gguf \
    --temp 1.0 \
    --min-p 0.01 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --seed 3407
```

{% endcode %}

6. As an example try: "Create a Flappy Bird game in HTML", and you will get:

{% columns %}
{% column width="33.33333333333333%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwgHfM2RE4JaK2shbPLWj%2Fimage.png?alt=media&amp;token=ae7e5bae-b4b5-45d8-a126-adc3695e36ad" alt="" width="188"><figcaption></figcaption></figure>
{% endcolumn %}

{% column width="66.66666666666667%" %}
{% code expandable="true" %}

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Flappy Bird</title>
    <style>
        body {
            margin: 0;
            padding: 0;
            background: #222;
            display: flex;
            justify-content: center;
            align-items: center;
            height: 100vh;
            font-family: 'Segoe UI', sans-serif;
            overflow: hidden;
            touch-action: none;
        }
        
        #game-container {
            position: relative;
            width: 400px;
            height: 600px;
            background: linear-gradient(to bottom, #70c5ce 0%, #70c5ce 80%, #c23810 80%, #c23810 100%);
            box-shadow: 0 0 20px rgba(0,0,0,0.5);
            overflow: hidden;
        }
        
        canvas {
            display: block;
        }
        
        .overlay {
            position: absolute;
            top: 50%;
            left: 50%;
            transform: translate(-50%, -50%);
            text-align: center;
            color: white;
            text-shadow: 2px 2px 0 #000;
            font-weight: bold;
            pointer-events: none;
        }
        
        .game-title {
            font-size: 48px;
            margin-bottom: 20px;
        }
        
        .score-display {
            font-size: 36px;
            margin-bottom: 10px;
        }
        
        .best-score {
            font-size: 24px;
            color: #ffe;
        }
        
        .instruction {
            font-size: 20px;
            animation: pulse 1s infinite;
        }
        
        @keyframes pulse {
            0%, 100% { opacity: 1; }
            50% { opacity: 0.5; }
        }
        
        .hidden { display: none; }
    </style>
</head>
<body>
    <div id="game-container">
        <canvas id="canvas" width="400" height="600"></canvas>
        
        <!-- Start Screen -->
        <div id="start-screen" class="overlay">
            <div class="game-title">FLAPPY BIRD</div>
            <div class="instruction">Click or Space to Fly</div>
        </div>
        
        <!-- Game Over Screen -->
        <div id="game-over-screen" class="overlay hidden">
            <div class="game-title">GAME OVER</div>
            <div class="score-display">Score: <span id="final-score">0</span></div>
            <div class="best-score">Best: <span id="best-score">0</span></div>
            <div class="instruction">Click to Restart</div>
        </div>
        
        <!-- Score Counter -->
        <div id="current-score" class="overlay hidden" style="top: 10%; font-size: 72px; color: white; text-shadow: 4px 4px 0 #000;">
            0
        </div>
    </div>

    <script>
        const canvas = document.getElementById('canvas');
        const ctx = canvas.getContext('2d');
        
        // Game constants
        const GRAVITY = 0.4;
        const JUMP_STRENGTH = -7;
        const PIPE_SPEED = 3;
        const PIPE_SPAWN_RATE = 120; // frames
        const PIPE_GAP = 120;
        
        // Game state
        let bird = { x: 50, y: 200, velocity: 0, radius: 15, wingState: 0 };
        let pipes = [];
        let score = 0;
        let bestScore = localStorage.getItem('flappyBest') || 0;
        let frameCount = 0;
        let isGameOver = false;
        let isPlaying = false;
        
        // DOM elements
        const startScreen = document.getElementById('start-screen');
        const gameOverScreen = document.getElementById('game-over-screen');
        const currentScoreDisplay = document.getElementById('current-score');
        const finalScoreEl = document.getElementById('final-score');
        const bestScoreEl = document.getElementById('best-score');
        
        // Input handling
        function handleInput(e) {
            if (!isPlaying) {
                if (isGameOver) {
                    resetGame();
                }
                startGame();
            } else if (!isGameOver) {
                bird.velocity = JUMP_STRENGTH;
                bird.wingState = 1;
            }
        }
        
        document.addEventListener('keydown', (e) => {
            if (e.code === 'Space' || e.code === 'ArrowUp') handleInput(e);
        });
        canvas.addEventListener('pointerdown', handleInput);
        
        function startGame() {
            isPlaying = true;
            isGameOver = false;
            startScreen.classList.add('hidden');
            currentScoreDisplay.classList.remove('hidden');
            resetGameState();
            gameLoop();
        }
        
        function resetGameState() {
            bird = { x: 50, y: 200, velocity: 0, radius: 15, wingState: 0 };
            pipes = [];
            score = 0;
            frameCount = 0;
            currentScoreDisplay.textContent = score;
        }
        
        function resetGame() {
            isGameOver = false;
            isPlaying = true;
            gameOverScreen.classList.add('hidden');
            currentScoreDisplay.classList.remove('hidden');
            resetGameState();
            gameLoop();
        }
        
        function spawnPipe() {
            const minHeight = 100;
            const maxHeight = 400;
            const topHeight = Math.floor(Math.random() * (maxHeight - minHeight + 1) + minHeight);
            const bottomHeight = canvas.height - topHeight - PIPE_GAP;
            
            pipes.push({
                x: canvas.width,
                topHeight: topHeight,
                bottomY: topHeight + PIPE_GAP,
                bottomHeight: bottomHeight,
                passed: false
            });
        }
        
        function update() {
            if (isGameOver) return;
            
            // Bird physics
            bird.velocity += GRAVITY;
            bird.y += bird.velocity;
            
            // Floor/ceiling collision
            if (bird.y + bird.radius > canvas.height || bird.y - bird.radius < 0) {
                gameOver();
                return;
            }
            
            // Pipe spawning
            frameCount++;
            if (frameCount % PIPE_SPAWN_RATE === 0) {
                spawnPipe();
            }
            
            // Pipe movement and collision
            for (let i = pipes.length - 1; i >= 0; i--) {
                const pipe = pipes[i];
                pipe.x -= PIPE_SPEED;
                
                // Remove off-screen pipes
                if (pipe.x + 60 < 0) {
                    pipes.splice(i, 1);
                    continue;
                }
                
                // Collision detection (simplified rect-circle)
                const pipeWidth = 60;
                const pipeX = pipe.x;
                const pipeLeft = pipeX;
                const pipeRight = pipeX + pipeWidth;
                
                // Bird is circle, pipes are rects
                const birdLeft = bird.x - bird.radius + 4; // +4 for beak offset
                const birdRight = bird.x + bird.radius + 2;
                const birdTop = bird.y - bird.radius;
                const birdBottom = bird.y + bird.radius;
                
                // Horizontal collision check
                if (birdRight > pipeLeft && birdLeft < pipeRight) {
                    // Top pipe collision
                    if (birdTop < pipe.topHeight) {
                        gameOver();
                        return;
                    }
                    // Bottom pipe collision
                    if (birdBottom > pipe.bottomY) {
                        gameOver();
                        return;
                    }
                }
                
                // Score counting
                if (pipe.x + pipeWidth < bird.x && !pipe.passued) {
                    pipe.passed = true;
                    score++;
                    currentScoreDisplay.textContent = score;
                }
            }
            
            // Animate wings
            if (bird.wingState > 0) {
                bird.wingState = (bird.wingState + 0.2) % 2;
            }
        }
        
        function draw() {
            // Clear canvas
            ctx.clearRect(0, 0, canvas.width, canvas.height);
            
            // Draw pipes
            pipes.forEach(pipe => {
                // Top pipe
                ctx.fillStyle = '#46c';
                ctx.fillRect(pipe.x, 0, 60, pipe.topHeight);
                ctx.fillStyle = '#34a';
                ctx.fillRect(pipe.x, pipe.topHeight - 20, 60, 20); // Cap
                
                // Bottom pipe
                ctx.fillStyle = '#46c';
                ctx.fillRect(pipe.x, pipe.bottomY, 60, canvas.height - pipe.bottomY);
                ctx.fillStyle = '#34a';
                ctx.fillRect(pipe.x, pipe.bottomY - 20, 60, 20); // Cap
            });
            
            // Draw bird (circle with beak)
            ctx.fillStyle = '#e3bc4e';
            ctx.beginPath();
            ctx.arc(bird.x, bird.y, bird.radius, 0, Math.PI * 2);
            ctx.fill();
            
            // Beak
            ctx.fillStyle = '#e04c4c';
            ctx.beginPath();
            ctx.moveTo(bird.x + bird.radius - 4, bird.y - 4);
            ctx.lineTo(bird.x + bird.radius + 10, bird.y);
            ctx.lineTo(bird.x + bird.radius - 4, bird.y + 4);
            ctx.fill();
            
            // Eyes
            ctx.fillStyle = 'black';
            ctx.beginPath();
            ctx.arc(bird.x + 5, bird.y - 6, 3, 0, Math.PI * 2);
            ctx.fill();
            
            // Wings
            ctx.fillStyle = '#c4a';
            ctx.beginPath();
            ctx.ellipse(bird.x - 5, bird.y + 5, 10, 6, 0, 0, Math.PI * 2);
            ctx.fill();
        }
        
        function gameOver() {
            isGameOver = true;
            isPlaying = false;
            
            // Update best score
            if (score > bestScore) {
                bestScore = score;
                localStorage.setItem('flappyBest', bestScore);
            }
            
            // Show game over screen
            currentScoreDisplay.classList.add('hidden');
            gameOverScreen.classList.remove('hidden');
            finalScoreEl.textContent = score;
            bestScoreEl.textContent = bestScore;
        }
        
        function gameLoop() {
            if (!isPlaying) return;
            
            update();
            draw();
            requestAnimationFrame(gameLoop);
        }
        
        // Initial draw
        draw();
    </script>
</body>
</html>
```

{% endcode %}
{% endcolumn %}
{% endcolumns %}

### ✨ Deploy with llama-server and OpenAI's completion library

{% hint style="success" %}
Using `--kv-unified` can make inference serving faster in llama.cpp! See <https://www.reddit.com/r/LocalLLaMA/comments/1qnwa33/glm_47_flash_huge_performance_improvement_with_kvu/>
{% endhint %}

After installing llama.cpp as per [#run-kimi-k2-thinking-in-llama.cpp](#run-kimi-k2-thinking-in-llama.cpp "mention"), you can use the below to launch an OpenAI compatible server:

{% code overflow="wrap" %}

```bash
LLAMA_SET_ROWS=1 ./llama.cpp/llama-server \
    --model unsloth/Kimi-K2.5-GGUF/UD-TQ1_0/Kimi-K2.5-UD-TQ1_0-00001-of-00005.gguf \
    --special \
    --alias "unsloth/Kimi-K2.5" \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --kv-unified
```

{% endcode %}

Then use OpenAI's Python library after `pip install openai` :

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Kimi-K2.5",
    messages = [{"role": "user", "content": "What is 1+1?"},],
)
print(completion.choices[0].message.content)
```

And we get:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkltZYBKtXcy5TzKKvGzN%2Fimage.png?alt=media&amp;token=d0a7c6e2-bac8-4d95-a5fa-446975dff581" alt="" width="563"><figcaption></figcaption></figure>

And in the other llama-server screen:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FjwlPZF5cdLHaHYgOhna3%2Fimage.png?alt=media&amp;token=602f1cbf-80bf-49ce-bbdb-21862d3d65fe" alt="" width="563"><figcaption></figcaption></figure>

### 📊 Benchmarks

You can view further below for benchmarks in table format:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FxRTS9YSfD8G0d9KiXK3W%2Fkimi%20k25%20benchmarks.jpg?alt=media&amp;token=b537ff73-136e-4bc6-ba76-0882ee30c72c" alt="" width="375"><figcaption></figcaption></figure>

#### Reasoning & Knowledge

| Benchmark           | Kimi K2.5 | GPT-5.2 | Claude 4.5 Opus | Gemini 3 Pro | DeepSeek V3.2 | Qwen3-VL-235B-A22B-Thinking |
| ------------------- | --------: | ------: | --------------: | -----------: | ------------: | --------------------------: |
| HLE-Full            |      30.1 |    34.5 |            30.8 |         37.5 |         25.1† |                           - |
| HLE-Full (w/ tools) |      50.2 |    45.5 |            43.2 |         45.8 |         40.8† |                           - |
| AIME 2025           |      96.1 |     100 |            92.8 |         95.0 |          93.1 |                           - |
| HMMT 2025 (Feb)     |      95.4 |    99.4 |          92.9\* |       97.3\* |          92.5 |                           - |
| IMO-AnswerBench     |      81.8 |    86.3 |          78.5\* |       83.1\* |          78.3 |                           - |
| GPQA-Diamond        |      87.6 |    92.4 |            87.0 |         91.9 |          82.4 |                           - |
| MMLU-Pro            |      87.1 |  86.7\* |          89.3\* |         90.1 |          85.0 |                           - |

#### Image & Video

| Benchmark            | Kimi K2.5 | GPT-5.2 | Claude 4.5 Opus | Gemini 3 Pro | DeepSeek V3.2 | Qwen3-VL-235B-A22B-Thinking |
| -------------------- | --------: | ------: | --------------: | -----------: | ------------: | --------------------------: |
| MMMU-Pro             |      78.5 |  79.5\* |            74.0 |         81.0 |             - |                        69.3 |
| CharXiv (RQ)         |      77.5 |    82.1 |          67.2\* |         81.4 |             - |                        66.1 |
| MathVision           |      84.2 |    83.0 |          77.1\* |       86.1\* |             - |                        74.6 |
| MathVista (mini)     |      90.1 |  82.8\* |          80.2\* |       89.8\* |             - |                        85.8 |
| ZeroBench            |         9 |     9\* |             3\* |          8\* |             - |                         4\* |
| ZeroBench (w/ tools) |        11 |     7\* |             9\* |         12\* |             - |                         3\* |
| OCRBench             |      92.3 |  80.7\* |          86.5\* |       90.3\* |             - |                        87.5 |
| OmniDocBench 1.5     |      88.8 |    85.7 |          87.7\* |         88.5 |             - |                      82.0\* |
| InfoVQA (val)        |      92.6 |    84\* |          76.9\* |       57.2\* |             - |                        89.5 |
| SimpleVQA            |      71.2 |  55.8\* |          69.7\* |       69.7\* |             - |                      56.8\* |
| WorldVQA             |      46.3 |    28.0 |            36.8 |         47.4 |             - |                        23.5 |
| VideoMMMU            |      86.6 |    85.9 |          84.4\* |         87.6 |             - |                        80.0 |
| MMVU                 |      80.4 |  80.8\* |            77.3 |         77.5 |             - |                        71.1 |
| MotionBench          |      70.4 |    64.8 |            60.3 |         70.3 |             - |                           - |
| VideoMME             |      87.4 |  86.0\* |               - |       88.4\* |             - |                        79.0 |
| LongVideoBench       |      79.8 |  76.5\* |          67.2\* |       77.7\* |             - |                      65.6\* |
| LVBench              |      75.9 |       - |               - |       73.5\* |             - |                        63.6 |

#### Coding

| Benchmark              | Kimi K2.5 | GPT-5.2 | Claude 4.5 Opus | Gemini 3 Pro | DeepSeek V3.2 | Qwen3-VL-235B-A22B-Thinking |
| ---------------------- | --------: | ------: | --------------: | -----------: | ------------: | --------------------------: |
| SWE-Bench Verified     |      76.8 |    80.0 |            80.9 |         76.2 |          73.1 |                           - |
| SWE-Bench Pro          |      50.7 |    55.6 |          55.4\* |            - |             - |                           - |
| SWE-Bench Multilingual |      73.0 |    72.0 |            77.5 |         65.0 |          70.2 |                           - |
| Terminal Bench 2.0     |      50.8 |    54.0 |            59.3 |         54.2 |          46.4 |                           - |
| PaperBench             |      63.5 |  63.7\* |          72.9\* |            - |          47.1 |                           - |
| CyberGym               |      41.3 |       - |            50.6 |       39.9\* |        17.3\* |                           - |
| SciCode                |      48.7 |    52.1 |            49.5 |         56.1 |          38.9 |                           - |
| OJBench (cpp)          |      57.4 |       - |          54.6\* |       68.5\* |        54.7\* |                           - |
| LiveCodeBench (v6)     |      85.0 |       - |          82.2\* |       87.4\* |          83.3 |                           - |

#### Long Context

| Benchmark    | Kimi K2.5 | GPT-5.2 | Claude 4.5 Opus | Gemini 3 Pro | DeepSeek V3.2 | Qwen3-VL-235B-A22B-Thinking |
| ------------ | --------: | ------: | --------------: | -----------: | ------------: | --------------------------: |
| Longbench v2 |      61.0 |  54.5\* |          64.4\* |       68.2\* |        59.8\* |                           - |
| AA-LCR       |      70.0 |  72.3\* |          71.3\* |       65.3\* |        64.3\* |                           - |

#### Agentic Search

| Benchmark                        | Kimi K2.5 | GPT-5.2 | Claude 4.5 Opus | Gemini 3 Pro | DeepSeek V3.2 | Qwen3-VL-235B-A22B-Thinking |
| -------------------------------- | --------: | ------: | --------------: | -----------: | ------------: | --------------------------: |
| BrowseComp                       |      60.6 |    65.8 |            37.0 |         37.8 |          51.4 |                           - |
| BrowseComp (w/ctx manage)        |      74.9 |    65.8 |            57.8 |         59.2 |          67.6 |                           - |
| BrowseComp (Agent Swarm)         |      78.4 |       - |               - |            - |             - |                           - |
| WideSearch (item-f1)             |      72.7 |       - |          76.2\* |         57.0 |        32.5\* |                           - |
| WideSearch (item-f1 Agent Swarm) |      79.0 |       - |               - |            - |             - |                           - |
| DeepSearchQA                     |      77.1 |  71.3\* |          76.1\* |       63.2\* |        60.9\* |                           - |
| FinSearchCompT2\&T3              |      67.8 |       - |          66.2\* |         49.9 |        59.1\* |                           - |
| Seal-0                           |      57.4 |    45.0 |          47.7\* |       45.5\* |        49.5\* |                           - |

#### Notes

* `*` = score re-evaluated by the authors (not publicly available previously).
* `†` = DeepSeek V3.2 score corresponds to its text-only subset (as noted in the footnotes).
* `-` = not evaluated / not available.


# GLM-4.7-Flash: How To Run Locally

Run & fine-tune GLM-4.7-Flash locally on your device!

GLM-4.7-Flash is Z.ai’s new 30B MoE reasoning model built for local deployment, delivering best-in-class performance for coding, agentic workflows, and chat. It uses \~3.6B parameters, supports 200K context, and leads SWE-Bench, GPQA, and reasoning/chat benchmarks.

GLM-4.7-Flash runs on **24GB RAM**/VRAM/unified memory (32GB for full precision), and you can now fine-tune with Unsloth. To run GLM 4.7 Flash with vLLM, see [#glm-4.7-flash-in-vllm](#glm-4.7-flash-in-vllm "mention")

{% hint style="success" %}
Jan 21 update: `llama.cpp` fixed a bug specifying the wrong `scoring_func`: `"softmax"` (should be `"sigmoid"`). This caused looping and poor outputs. We updated the GGUFs - please re-download the model for much better outputs.

You can now use Z.ai’s recommended parameters and get great results:

* **For general use-case:** `--temp 1.0 --top-p 0.95`
* **For tool-calling:** `--temp 0.7 --top-p 1.0`
* **Repeat penalty:** Disable it, or set `--repeat-penalty 1.0`

Jan 22: Faster inference is here as the FA fix for CUDA is now merged.
{% endhint %}

<a href="/pages/zWg6fYWpGDwkPXd58ReM#run-glm-4.7-flash" class="button primary">Running Tutorial</a><a href="/pages/zWg6fYWpGDwkPXd58ReM#fine-tuning-glm-4.7-flash" class="button secondary">Fine-tuning</a>

GLM-4.7-Flash GGUF to run: [unsloth/GLM-4.7-Flash-GGUF](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF)

### ⚙️ Usage Guide

For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower.

After speaking with the Z.ai's team, they recommend using their GLM-4.7 sampling parameters:

| Default Settings (Most Tasks)                                      | Terminal Bench, SWE Bench Verified                                 |
| ------------------------------------------------------------------ | ------------------------------------------------------------------ |
| <mark style="background-color:green;">**temperature = 1.0**</mark> | <mark style="background-color:green;">**temperature = 0.7**</mark> |
| <mark style="background-color:green;">**top\_p = 0.95**</mark>     | <mark style="background-color:green;">**top\_p = 1.0**</mark>      |
| repeat penalty = disabled or 1.0                                   | repeat penalty = disabled or 1.0                                   |

* For general use-case:  `--temp 1.0 --top-p 0.95`
* For tool-calling:  `--temp 0.7 --top-p 1.0`
* If using llama.cpp, set `--min-p 0.01` as llama.cpp's default is 0.05
* Sometimes you'll need to experiment what numbers work best for your use-case.

{% hint style="warning" %}
For now, we **don’t recommend** running this GGUF with **Ollama** due to potential chat template compatibility issues. The GGUF works well on llama.cpp (or backends e.g. LM Studio, Jan).

**Remember to disable repeat penalty! Or set** `--repeat-penalty 1.0`
{% endhint %}

* **Maximum context window:** `202,752`

### 🖥️ Run GLM-4.7-Flash

Depending on your use-case you will need to use different settings. Some GGUFs end up similar in size because the model architecture (like [gpt-oss](/docs/models/gpt-oss-how-to-run-and-fine-tune)) has dimensions not divisible by 128, so parts can’t be quantized to lower bits.

Because this guide uses 4-bit, you will need around 18GB RAM/unified memory. We recommend using at least 4-bit precision for best performance.

{% hint style="warning" %}
For now, we **don’t recommend** running this GGUF with **Ollama** due to potential chat template compatibility issues. The GGUF works well on llama.cpp (or backends e.g. LM Studio, Jan).

**Remember to disable repeat penalty! Or set** `--repeat-penalty 1.0`
{% endhint %}

#### 🦥 Unsloth Studio Guide

GLM-4.7-Flash can be run and fine-tuned in [Unsloth Studio](/docs/new/studio), our new open-source web UI for local AI. With Unsloth Studio, you can run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio#run-models-locally) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFeQ0UUlnjXkDdqhcWglh%2Fskinny%20studio%20chat.png?alt=media&amp;token=c2ee045f-c243-4024-a8e4-bb4dbe7bae79" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}
**Install Unsloth**

Run in your terminal:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% hint style="success" %}
**Installation will be quick and take approx 1-2 mins.**
{% endhint %}
{% endstep %}

{% step %}
**Launch Unsloth**

MacOS, Linux, WSL and Windows:

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

Then open `http://localhost:8888` in your browser.
{% endstep %}

{% step %}
**Search and download GLM-4.7-Flash**

On first launch you will need to create a password to secure your account and sign in again later. You’ll then see a brief onboarding wizard to choose a model, dataset, and basic settings. You can skip it at any time.

Then go to the [Unsloth Chat](/docs/new/studio/chat) tab and search for **GLM-4.7-Flash** in the search bar and download your desired model and quant.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F4oQKFbYhEz7h2K9H3EuP%2FScreenshot%202026-04-02%20at%201.59.45%E2%80%AFAM.png?alt=media&amp;token=bf78de14-cbcb-4f99-b9b4-65a500a6dd20" alt="" width="375"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}
**Run GLM-4.7-Flash**

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth Studio inference guide](/docs/new/studio/chat).

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FXPQGEEr1YoKofrTatAKK%2Ftoolcallingif.gif?alt=media&amp;token=25d68698-fb13-4c46-99b2-d39fb025df08" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}
{% endstepper %}

#### Llama.cpp Tutorial (GGUF):

Instructions to run in llama.cpp (note we will be using 4-bit to fit most devices):

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
You can directly pull from Hugging Face. You can increase the context to 200K as your RAM/VRAM allows.

You can also try Z.ai's recommended GLM-4.7 sampling parameters:

* For general use-case:  `--temp 1.0 --top-p 0.95`
* For tool-calling:  `--temp 0.7 --top-p 1.0`
* **Remember to disable repeat penalty!**

Follow this for **general instruction** use-cases:

```bash
./llama.cpp/llama-cli \
    -hf unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 0.95 --min-p 0.01
```

Follow this for **tool-calling** use-cases:

```bash
./llama.cpp/llama-cli \
    -hf unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 --top-p 1.0 --min-p 0.01
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub`). You can choose `UD-Q4_K_XL` or other quantized versions. If downloads get stuck, see [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

{% code overflow="wrap" %}

```bash
pip install -U huggingface_hub
hf download unsloth/GLM-4.7-Flash-GGUF \
    --local-dir unsloth/GLM-4.7-Flash-GGUF \
    --include "*UD-Q2_K_XL*"
```

{% endcode %}
{% endstep %}

{% step %}
Then run the model in conversation mode:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01
```

{% endcode %}

Also, adjust **context window** as required, up to `202752`
{% endstep %}
{% endstepper %}

### :loop:Reducing repetition and looping

{% hint style="success" %}
**JAN 21 UPDATE: llama.cpp fixed a bug specifying the wrong** `"scoring_func": "softmax"` **that caused looping and poor outputs (should be sigmoid) We updated the GGUFs. Please re-download the model for much better outputs.**
{% endhint %}

This means you can now use Z.ai's recommended parameters and get great results:

* For general use-case:  `--temp 1.0 --top-p 0.95`
* For tool-calling:  `--temp 0.7 --top-p 1.0`
* If using llama.cpp, set `--min-p 0.01` as llama.cpp's default is 0.05
* **Remember to disable repeat penalty! Or set** `--repeat-penalty 1.0`

We added `"scoring_func": "sigmoid"` to `config.json` for the main model - [see](https://huggingface.co/unsloth/GLM-4.7-Flash/commit/3fd53b491e04f707f307aef2f70f8a7520511e6d).

{% hint style="warning" %}
For now, we **don’t recommend** running this GGUF with **Ollama** due to potential chat template compatibility issues. The GGUF works well on llama.cpp (or backends e.g. LM Studio, Jan).
{% endhint %}

### :bird:Flappy Bird Example with UD-Q4\_K\_XL

As an example, we did the following long conversation by using UD-Q4\_K\_XL via `./llama.cpp/llama-cli --model unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf --fit on --temp 1.0 --top-p 0.95 --min-p 0.01` :

```
Hi
What is 2+2
Create a Python Flappy Bird game
Create a totally different game in Rust
Find bugs in both
Make the 1st game I mentioned but in a standalone HTML file
Find bugs and show the fixed game
```

which rendered the following Flappy Bird game in HTML form:

<details>

<summary>Flappy Bird Game in HTML (Expandable)</summary>

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
    <title>Flappy Bird Fixed</title>
    <style>
        body {
            margin: 0;
            display: flex;
            justify-content: center;
            align-items: center;
            height: 100vh;
            background-color: #222;
            font-family: 'Arial', sans-serif;
            overflow: hidden;
            user-select: none;
            -webkit-user-select: none;
            touch-action: none; /* Prevents zoom on mobile */
        }

        #game-container {
            position: relative;
            box-shadow: 0 0 20px rgba(0,0,0,0.5);
        }

        canvas {
            background-color: #87CEEB;
            display: block;
            border-radius: 4px;
        }

        /* UI Overlays */
        #ui-layer {
            position: absolute;
            top: 0;
            left: 0;
            width: 100%;
            height: 100%;
            pointer-events: none;
            display: flex;
            flex-direction: column;
            justify-content: center;
            align-items: center;
            text-align: center;
        }

        #score-display {
            position: absolute;
            top: 40px;
            left: 50%;
            transform: translateX(-50%);
            font-size: 48px;
            font-weight: bold;
            color: white;
            text-shadow: 3px 3px 0 #000;
            z-index: 10;
            font-family: 'Courier New', Courier, monospace;
        }

        #start-screen, #game-over-screen {
            background: rgba(0, 0, 0, 0.7);
            width: 100%;
            height: 100%;
            display: flex;
            flex-direction: column;
            justify-content: center;
            align-items: center;
            color: white;
            pointer-events: auto; /* Allow clicks */
            cursor: pointer;
        }

        h1 { margin: 0 0 10px 0; font-size: 60px; text-shadow: 4px 4px 0 #000; line-height: 1; }
        p { font-size: 22px; margin: 10px 0; color: #ddd; }
        
        .btn {
            background: linear-gradient(to bottom, #ffeb3b, #fbc02d);
            border: 3px solid #fff;
            color: #333;
            padding: 15px 40px;
            font-size: 28px;
            font-weight: bold;
            cursor: pointer;
            border-radius: 8px;
            box-shadow: 0 6px 0 #c49000, 0 10px 10px rgba(0,0,0,0.3);
            text-transform: uppercase;
            transition: all 0.1s;
            margin-top: 10px;
        }

        .btn:active {
            transform: translateY(4px);
            box-shadow: 0 2px 0 #c49000, 0 4px 4px rgba(0,0,0,0.3);
        }

        .score-board {
            background: #ded895;
            border: 2px solid #543847;
            padding: 20px 40px;
            border-radius: 10px;
            box-shadow: 4px 4px 0 #543847;
            margin-bottom: 30px;
            display: none;
            border: 4px solid #543847;
        }
        
        .score-board h2 { margin: 0 0 5px 0; color: #e86101; font-size: 40px; }
        .score-board span { font-size: 20px; color: #543847; display: block; text-align: center; }

    </style>
</head>
<body>

    <div id="game-container">
        <canvas id="gameCanvas" width="400" height="600"></canvas>
        
        <div id="score-display">0</div>

        <div id="ui-layer">
            <div id="start-screen">
                <h1>FLAPPY<br>BIRD</h1>
                <p>Tap or Press Space to Start</p>
                <button class="btn" style="display:none;" id="touch-instruction">Click to Start</button>
            </div>

            <div id="game-over-screen">
                <h1>GAME OVER</h1>
                <div class="score-board" id="score-board">
                    <h2>Score: <span id="final-score">0</span></h2>
                </div>
                <button class="btn" id="restart-btn">Try Again</button>
            </div>
        </div>
    </div>

<script>
    const canvas = document.getElementById('gameCanvas');
    const ctx = canvas.getContext('2d');

    // --- Constants ---
    const GRAVITY = 0.35; // Slightly harder gravity for better feel
    const JUMP_STRENGTH = -6.5;
    const PIPE_GAP = 180;
    const PIPE_WIDTH = 60;
    const PIPE_SPEED = 2.5;
    const PIPE_SPAWN_RATE = 100;

    // --- State ---
    let frames = 0;
    let score = 0;
    let isGameOver = false;
    let isPlaying = false;
    let gameLoopId;

    const ui = {
        startScreen: document.getElementById('start-screen'),
        gameOverScreen: document.getElementById('game-over-screen'),
        scoreDisplay: document.getElementById('score-display'),
        scoreBoard: document.getElementById('score-board'),
        finalScore: document.getElementById('final-score'),
        restartBtn: document.getElementById('restart-btn')
    };

    const bird = {
        x: 80,
        y: 150,
        radius: 12, // Fixed radius
        velocity: 0,
        
        draw: function() {
            // Rotate bird based on velocity for visual flair
            let angle = Math.min(Math.PI / 4, Math.max(-Math.PI / 4, (this.velocity * 0.1)));
            
            ctx.save();
            ctx.translate(this.x, this.y);
            ctx.rotate(angle);
            
            // Draw Body
            ctx.fillStyle = '#FFD700';
            ctx.beginPath();
            ctx.arc(0, 0, this.radius, 0, Math.PI * 2);
            ctx.fill();
            
            // Eye
            ctx.fillStyle = 'white';
            ctx.beginPath();
            ctx.arc(4, -4, 4, 0, Math.PI * 2);
            ctx.fill();
            ctx.fillStyle = 'black';
            ctx.beginPath();
            ctx.arc(6, -4, 2, 0, Math.PI * 2);
            ctx.fill();
            
            // Wing
            ctx.fillStyle = '#FFA500';
            ctx.beginPath();
            ctx.arc(-4, 4, 5, 0, Math.PI * 2);
            ctx.fill();

            ctx.restore();
        },

        update: function() {
            this.velocity += GRAVITY;
            this.y += this.velocity;
        },

        jump: function() {
            this.velocity = JUMP_STRENGTH;
        },

        reset: function() {
            this.y = 150;
            this.velocity = 0;
        }
    };

    let pipes = [];

    function createPipe() {
        const minHeight = 50;
        const maxPos = canvas.height - PIPE_GAP - minHeight;
        const topHeight = Math.floor(Math.random() * (maxPos - minHeight + 1)) + minHeight;
        
        pipes.push({
            x: canvas.width,
            topHeight: topHeight,
            bottomY: topHeight + PIPE_GAP,
            width: PIPE_WIDTH,
            passed: false
        });
    }

    function drawPipes() {
        ctx.fillStyle = '#2ecc71';
        ctx.strokeStyle = '#27ae60';
        ctx.lineWidth = 2;
        
        pipes.forEach(pipe => {
            // Top Pipe
            ctx.fillRect(pipe.x, 0, pipe.width, pipe.topHeight);
            ctx.strokeRect(pipe.x, 0, pipe.width, pipe.topHeight);
            
            // Bottom Pipe
            ctx.fillRect(pipe.x, pipe.bottomY, pipe.width, canvas.height - pipe.bottomY);
            ctx.strokeRect(pipe.x, pipe.bottomY, pipe.width, canvas.height - pipe.bottomY);

            // Cap
            const capH = 20;
            ctx.fillStyle = '#27ae60'; 
            ctx.fillRect(pipe.x - 2, pipe.topHeight - capH, pipe.width + 4, capH);
            ctx.fillRect(pipe.x - 2, pipe.bottomY, pipe.width + 4, capH);
        });
    }

    function updatePipes() {
        if (frames % PIPE_SPAWN_RATE === 0) createPipe();

        for (let i = 0; i < pipes.length; i++) {
            let p = pipes[i];
            p.x -= PIPE_SPEED;

            // --- FIXED COLLISION DETECTION ---
            // Treat bird as a circle of radius 'bird.radius'
            // Pipe is a rect: x, x+w, y_top, y_bottom
            let birdLeft = bird.x - bird.radius;
            let birdRight = bird.x + bird.radius;
            let birdTop = bird.y - bird.radius;
            let birdBottom = bird.y + bird.radius;

            // Horizontal Overlap
            if (birdRight > p.x && birdLeft < p.x + p.width) {
                // Vertical Overlap (Hit Top Pipe OR Hit Bottom Pipe)
                if (birdTop < p.topHeight || birdBottom > p.bottomY) {
                    gameOver();
                }
            }

            // --- FIXED SCORING ---
            // If pipe is off screen to the left, and hasn't been scored
            if (p.x + p.width < 0 && !p.passed) {
                score++;
                p.passed = true;
                ui.scoreDisplay.innerText = score;
            }

            if (p.x < -60) {
                pipes.shift();
                i--;
            }
        }
    }

    function checkCollisions() {
        // Floor
        if (bird.y + bird.radius >= canvas.height) {
            gameOver();
        }
        // Ceiling
        if (bird.y - bird.radius <= 0) {
            bird.y = bird.radius;
            bird.velocity = 0;
        }
    }

    function drawBackground() {
        // Clear
        ctx.clearRect(0, 0, canvas.width, canvas.height);
        
        // Floor
        ctx.fillStyle = '#654321';
        ctx.fillRect(0, canvas.height - 10, canvas.width, 10);
        
        // Clouds
        ctx.fillStyle = "rgba(255, 255, 255, 0.6)";
        for(let i=0; i<4; i++) {
            let x = (frames * 0.5 + i * 150) % (canvas.width + 100) - 50;
            let y = (i * 40) + 20;
            let scale = 1 + (Math.sin(frames * 0.02 + i) * 0.1);
            let size = 30 * scale;
            ctx.beginPath();
            ctx.arc(x, y, size, 0, Math.PI * 2);
            ctx.arc(x + 20*scale, y - 10*scale, size * 1.2, 0, Math.PI * 2);
            ctx.arc(x + 40*scale, y, size, 0, Math.PI * 2);
            ctx.fill();
        }
    }

    function update() {
        if (!isPlaying) return;
        bird.update();
        updatePipes();
        checkCollisions();
        frames++;
    }

    function draw() {
        drawBackground();
        drawPipes();
        bird.draw();
    }

    function loop() {
        update();
        draw();
        if (isPlaying || !isGameOver) {
            gameLoopId = requestAnimationFrame(loop);
        }
    }

    function startGame() {
        isPlaying = true;
        isGameOver = false;
        
        // UI
        ui.startScreen.style.display = 'none';
        ui.gameOverScreen.style.display = 'none';
        ui.scoreBoard.style.display = 'none';
        
        // Logic
        bird.reset();
        pipes = [];
        score = 0;
        frames = 0;
        ui.scoreDisplay.innerText = '0';
        
        loop();
    }

    function gameOver() {
        isPlaying = false;
        isGameOver = true;
        cancelAnimationFrame(gameLoopId);
        
        ui.finalScore.innerText = score;
        ui.gameOverScreen.style.display = 'flex';
        ui.scoreBoard.style.display = 'block';
    }

    // --- Input Handling ---

    function handleInput(e) {
        if (e.type === 'keydown' && e.code === 'Space') e.preventDefault();

        if (isPlaying) {
            bird.jump();
        } else if (!isGameOver) {
            // Click on start screen (or any click if game hasn't started)
            startGame();
        }
    }

    // Keyboard
    window.addEventListener('keydown', (e) => {
        if (e.code === 'Space') handleInput(e);
    });

    // Mouse / Touch
    window.addEventListener('mousedown', handleInput);
    window.addEventListener('touchstart', (e) => {
        // Prevent zoom/scroll
        // e.preventDefault(); 
        handleInput(e);
    }, {passive: false});

    // UI Interactions
    ui.restartBtn.addEventListener('click', (e) => {
        e.stopPropagation();
        startGame();
    });
    
    // Allow clicking the Game Over overlay to restart
    ui.gameOverScreen.addEventListener('mousedown', (e) => {
        if(e.target === ui.gameOverScreen) startGame();
    });
    ui.gameOverScreen.addEventListener('touchstart', (e) => {
        if(e.target === ui.gameOverScreen) {
            e.preventDefault();
            startGame();
        }
    });

    // Initial Draw
    drawBackground();
    bird.reset();
    bird.draw();

</script>
</body>
</html>
```

</details>

And we took some screenshots (4bit works):

<div align="left"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJ85uzHPDWinaXPe6kWyU%2Fimage.png?alt=media&amp;token=6547f49d-2544-4c48-a7d5-5c1c67d34a87" alt="" width="188"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHAc2SjNLo1OsyAC4dArm%2Fimage.png?alt=media&amp;token=87d4bfea-4ac9-41ef-be1c-1e51664d30b1" alt="" width="188"><figcaption></figcaption></figure></div>

### 🦥 Fine-tuning GLM-4.7-Flash

Unsloth now supports fine-tuning of GLM-4.7-Flash, however you will need to use `transformers v5`. The 30B model does not fit on a free Colab GPU; however, you can use our notebook. 16-bit LoRA fine-tuning of GLM-4.7-Flash will use around **60GB VRAM**:

* [GLM-4.7-Flash SFT LoRA notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GLM_Flash_A100\(80GB\).ipynb)

{% hint style="warning" %}
You may encounter out of memory sometimes when using A100 40GB VRAM. You will need to use H100/A100 80GB VRAM for smoother runs.
{% endhint %}

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/GLM_Flash_A100(80GB).ipynb>" %}

On fine-tuning MoE's, it's probably not a good idea to fine-tune the router layer so we disabled it by default. If you want to maintain its reasoning capabilities (optional), you can use a mix of direct answers and chain-of-thought examples. Use at least <mark style="background-color:green;">75% reasoning</mark> and <mark style="background-color:green;">25% non-reasoning</mark> in your dataset to make the model retain its reasoning capabilities.

### 🦙Llama-server serving & deployment

To deploy GLM-4.7-Flash for production, we use `llama-server` In a new terminal say via tmux, deploy the model via:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf \
    --alias "unsloth/GLM-4.7-Flash" \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001
```

{% endcode %}

Then in a new terminal, after doing `pip install openai`, do:

{% code overflow="wrap" %}

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/GLM-4.7-Flash",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
```

{% endcode %}

Which will print

{% code overflow="wrap" %}

```
User asks a simple question: "What is 2+2?" The answer is 4. Provide answer.

2 + 2 = 4.
```

{% endcode %}

### :computer: GLM-4.7-Flash in vLLM

You can now use our new [FP8 Dynamic quant](https://huggingface.co/unsloth/GLM-4.7-Flash-FP8-Dynamic) of the model for premium and fast inference. First install vLLM from nightly:

{% code overflow="wrap" %}

```bash
uv pip install --upgrade --force-reinstall vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly/cu130
uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git
uv pip install --force-reinstall numba
```

{% endcode %}

Then serve [Unsloth's dynamic FP8 version](https://huggingface.co/unsloth/GLM-4.7-Flash-FP8-Dynamic) of the model. We enabled FP8 to reduce KV cache memory usage by 50%, and on 4 GPUs. If you have 1 GPU, use `CUDA_VISIBLE_DEVICES='0'` and set `--tensor-parallel-size 1` or remove this argument. To disable FP8, remove `--quantization fp8 --kv-cache-dtype fp8`

```bash
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
CUDA_VISIBLE_DEVICES='0,1,2,3' vllm serve unsloth/GLM-4.7-Flash-FP8-Dynamic \
    --served-model-name unsloth/GLM-4.7-Flash \
    --tensor-parallel-size 4 \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice \
    --dtype bfloat16 \
    --seed 3407 \
    --max-model-len 200000 \
    --gpu-memory-utilization 0.95 \
    --max_num_batched_tokens 16384 \
    --port 8001 \
    --kv-cache-dtype fp8
```

You can then call the served model via the OpenAI API:

```python
from openai import AsyncOpenAI, OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8001/v1"
client = OpenAI( # or AsyncOpenAI
    api_key=openai_api_key,
    base_url=openai_api_base,
)
```

#### :star: vLLM GLM-4.7-Flash Speculative Decoding

We found using the MTP (multi token prediction) module from GLM 4.7 Flash makes generation throughput drop from 13,000 tokens on 1 B200 to 1,300 tokens! (10x slower) On Hopper, it should be fine hopefully.

```bash
    --speculative-config.method mtp \
    --speculative-config.num_speculative_tokens 1
```

Only 1,300 tokens / s throughput on 1xB200 (130 tokens / s decoding per user)

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FCJPJYh1uIS1yL8vskiOF%2Fimage.png?alt=media&amp;token=f07aaad9-93bd-4507-836f-967a3d39b0e5" alt=""><figcaption></figcaption></figure>

And 13,000 tokens / s throughput  on 1xB200 (still 130 token /s decoding per user)

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrXaHEknUb1QW1v0arO0q%2Fimage.png?alt=media&amp;token=dd81b731-90bb-4d1b-a647-a64618f5952a" alt=""><figcaption></figcaption></figure>

### :hammer:Tool Calling with GLM-4.7-Flash

See [Tool Calling Guide](/docs/basics/tool-calling-guide-for-local-llms) for more details on how to do tool calling. In a new terminal (if using tmux, use CTRL+B+D), we create some tools like adding 2 numbers, executing Python code, executing Linux functions and much more:

{% code expandable="true" %}

```python
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def subtract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([
        "A long time ago in a galaxy far far away...",
        "There were 2 friends who loved sloths and code...",
        "The world was ending because every sloth evolved to have superhuman intelligence...",
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "subtract_number": subtract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [
    {
        "type": "function",
        "function": {
            "name": "add_number",
            "description": "Add two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "multiply_number",
            "description": "Multiply two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "subtract_number",
            "description": "Subtract two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "write_a_story",
            "description": "Writes a random story.",
            "parameters": {
                "type": "object",
                "properties": {},
                "required": [],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "terminal",
            "description": "Perform operations from the terminal.",
            "parameters": {
                "type": "object",
                "properties": {
                    "command": {
                        "type": "string",
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",
                    },
                },
                "required": ["command"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "python",
            "description": "Call a Python interpreter with some Python code that will be ran.",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {
                        "type": "string",
                        "description": "The Python code to run",
                    },
                },
                "required": ["code"],
            },
        },
    },
]
```

{% endcode %}

We then use the below functions (copy and paste and execute) which will parse the function calls automatically and call the OpenAI endpoint for any model:

{% code overflow="wrap" expandable="true" %}

```python
from openai import OpenAI
def unsloth_inference(
    messages,
    temperature = 0.7,
    top_p = 1.0,
    top_k = -1,
    repetition_penalty = 0.0,
):
    messages = messages.copy()
    openai_client = OpenAI(
        base_url = "http://127.0.0.1:8001/v1",
        api_key = "sk-no-key-required",
    )
    model_name = next(iter(openai_client.models.list())).id
    print(f"Using model = {model_name}")
    has_tool_calls = True
    original_messages_len = len(messages)
    while has_tool_calls:
        print(f"Current messages = {messages}")
        response = openai_client.chat.completions.create(
            model = model_name,
            messages = messages,
            temperature = temperature,
            top_p = top_p,
            tools = tools if tools else None,
            tool_choice = "auto" if tools else None,
            extra_body = {"top_k": top_k, "min_p": min_p, "dry_multiplier" :repetition_penalty,}
        )
        tool_calls = response.choices[0].message.tool_calls or []
        content = response.choices[0].message.content or ""
        tool_calls_dict = [tc.to_dict() for tc in tool_calls] if tool_calls else tool_calls
        messages.append({"role": "assistant", "tool_calls": tool_calls_dict, "content": content,})
        for tool_call in tool_calls:
            fx, args, _id = tool_call.function.name, tool_call.function.arguments, tool_call.id
            out = MAP_FN[fx](**json.loads(args))
            messages.append({"role": "tool", "tool_call_id": _id, "name": fx, "content": str(out),})
        else:
            has_tool_calls = False
    return messages
```

{% endcode %}

After launching GLM-4.7-Flash via `llama-server` like in [#deploy-with-llama-server-and-openais-completion-library](#deploy-with-llama-server-and-openais-completion-library "mention") or see [Tool Calling Guide](/docs/basics/tool-calling-guide-for-local-llms) for more details, we then can do some tool calls:

**Tool Call for mathematical operations for GLM 4.7**

{% code overflow="wrap" %}

```python
messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "What is today's date plus 3 days?"}],
}]
unsloth_inference(messages, temperature = 1.0, top_p = 0.95, top_k = -1, min_p = 0.01)
```

{% endcode %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoFkZ20QOSGdzT4iz2SOB%2Fimage.png?alt=media&amp;token=e4ca30b0-dcec-4a26-b019-dd33f0600949" alt=""><figcaption></figcaption></figure>

**Tool Call to execute generated Python code for GLM-4.7-Flash**

{% code overflow="wrap" %}

```python
messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "Create a Fibonacci function in Python and find fib(20)."}],
}]
unsloth_inference(messages, temperature = 1.0, top_p = 0.95, top_k = -1, min_p = 0.01)
```

{% endcode %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhS8sWtZwjwerElezCc2C%2Fimage.png?alt=media&amp;token=39032ef8-386e-4837-8dd2-c552c80a3ee3" alt="" width="563"><figcaption></figcaption></figure>

### Benchmarks

GLM-4.7-Flash is the best performing 30B model across all benchmarks except AIME 25.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FuU8GDqR2ni6slcQ1pWzL%2Fglm4.7%20benchmarks.jpg?alt=media&amp;token=1f609b73-c20b-4e4c-bbf8-0e7213c00b44" alt="" width="375"><figcaption></figcaption></figure>

| Benchmark          | GLM-4.7-Flash | Qwen3-30B-A3B-Thinking-2507 | GPT-OSS-20B |
| ------------------ | ------------- | --------------------------- | ----------- |
| AIME 25            | 91.6          | 85.0                        | 91.7        |
| GPQA               | 75.2          | 73.4                        | 71.5        |
| LCB v6             | 64.0          | 66.0                        | 61.0        |
| HLE                | 14.4          | 9.8                         | 10.9        |
| SWE-bench Verified | 59.2          | 22.0                        | 34.0        |
| τ²-Bench           | 79.5          | 49.0                        | 47.7        |
| BrowseComp         | 42.8          | 2.29                        | 28.3        |


# Gemma 3 - How to Run Guide

How to run Gemma 3 effectively with our GGUFs on llama.cpp, Ollama, Open WebUI and how to fine-tune with Unsloth!

Google releases Gemma 3 with a new 270M model and the previous 1B, 4B, 12B, and 27B sizes. The 270M and 1B are text-only, while larger models handle both text and vision. We provide GGUFs, and a guide of how to run it effectively, and how to finetune & do [RL](/docs/get-started/reinforcement-learning-rl-guide) with Gemma 3!

{% hint style="success" %}
**NEW Aug 14, 2025 Update:** Try our fine-tuning [Gemma 3 (270M) notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(270M\).ipynb) and [GGUFs to run](https://huggingface.co/collections/unsloth/gemma-3-67d12b7e8816ec6efa7e4e5b).

Also see our [Gemma 3n Guide](/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune/gemma-3n-how-to-run-and-fine-tune).
{% endhint %}

<a href="#gmail-running-gemma-3-on-your-phone" class="button primary">Running Tutorial</a><a href="#fine-tuning-gemma-3-in-unsloth" class="button secondary">Fine-tuning Tutorial</a>

**Unsloth is the only framework which works in float16 machines for Gemma 3 inference and training.** This means Colab Notebooks with free Tesla T4 GPUs also work!

* Fine-tune Gemma 3 (4B) with vision support using our [free Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision.ipynb)

{% hint style="info" %}
According to the Gemma team, the optimal config for inference is\
`temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0`
{% endhint %}

**Unsloth Gemma 3 uploads with optimal configs:**

| GGUF                                                                                                                                                                                                                                                                                                                                                                                                                 | Unsloth Dynamic 4-bit Instruct                                                                                                                                                                                                                                                                                                                                                                                                                           | 16-bit Instruct                                                                                                                                                                                                                                                                                                                                                                 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <ul><li><a href="https://huggingface.co/unsloth/gemma-3-270m-it-GGUF">270M</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-1b-it-GGUF">1B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-4b-it-GGUF">4B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-12b-it-GGUF">12B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-27b-it-GGUF">27B-it</a></li></ul> | <ul><li><a href="https://huggingface.co/unsloth/gemma-3-270m-it-unsloth-bnb-4bit">270M</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-1b-it-bnb-4bit">1B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-4b-it-bnb-4bit">4B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-12b-it-unsloth-bnb-4bit">12B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-27b-it-bnb-4bit">27B-it</a></li></ul> | <ul><li><a href="https://huggingface.co/unsloth/gemma-3-270m-it">270M</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-1b">1B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-4b">4B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-12b">12B-it</a></li><li><a href="https://huggingface.co/unsloth/gemma-3-27b">27B-it</a></li></ul> |

## :gear: Recommended Inference Settings

According to the Gemma team, the official recommended settings for inference is:

* Temperature of 1.0
* Top\_K of 64
* Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
* Top\_P of 0.95
* Repetition Penalty of 1.0. (1.0 means disabled in llama.cpp and transformers)
* Chat template:

  <pre data-overflow="wrap"><code><strong>&#x3C;bos>&#x3C;start_of_turn>user\nHello!&#x3C;end_of_turn>\n&#x3C;start_of_turn>model\nHey there!&#x3C;end_of_turn>\n&#x3C;start_of_turn>user\nWhat is 1+1?&#x3C;end_of_turn>\n&#x3C;start_of_turn>model\n
  </strong></code></pre>
* Chat template with `\n`newlines rendered (except for the last)

{% code overflow="wrap" %}

```
<bos><start_of_turn>user
Hello!<end_of_turn>
<start_of_turn>model
Hey there!<end_of_turn>
<start_of_turn>user
What is 1+1?<end_of_turn>
<start_of_turn>model\n
```

{% endcode %}

{% hint style="danger" %}
llama.cpp an other inference engines auto add a \<bos> - DO NOT add TWO \<bos> tokens! You should ignore the \<bos> when prompting the model!
{% endhint %}

### ✨Running Gemma 3 on your phone <a href="#gmail-running-gemma-3-on-your-phone" id="gmail-running-gemma-3-on-your-phone"></a>

To run the models on your phone, we recommend using any mobile app that can run GGUFs locally on edge devices like phones. After fine-tuning you can export it to GGUF then run it locally on your phone. Ensure your phone has enough RAM/power to process the models as it can overheat so we recommend using Gemma 3 270M or the Gemma 3n models for this use-case. You can try the [open-source project AnythingLLM's](https://github.com/Mintplex-Labs/anything-llm) mobile app which you can download on [Android here](https://play.google.com/store/apps/details?id=com.anythingllm) or [ChatterUI](https://github.com/Vali-98/ChatterUI), which are great apps for running GGUFs on your phone.

{% hint style="success" %}
Remember, you can change the model name 'gemma-3-27b-it-GGUF' to any Gemma model like 'gemma-3-270m-it-GGUF:Q8\_K\_XL' for all the tutorials.
{% endhint %}

## :llama: Tutorial: How to Run Gemma 3 in Ollama

1. Install `ollama` if you haven't already!

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! You can change the model name 'gemma-3-27b-it-GGUF' to any Gemma model like 'gemma-3-270m-it-GGUF:Q8\_K\_XL'.

```bash
ollama run hf.co/unsloth/gemma-3-27b-it-GGUF:Q4_K_XL
```

## 📖 Tutorial: How to Run Gemma 3 27B in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run`

```bash
./llama.cpp/llama-mtmd-cli \
    -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL
```

3. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). More versions at: <https://huggingface.co/unsloth/gemma-3-27b-it-GGUF>

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/gemma-3-27b-it-GGUF",
    local_dir = "unsloth/gemma-3-27b-it-GGUF",
    allow_patterns = ["*Q4_K_XL*", "mmproj-BF16.gguf"], # For Q4_K_M
)
```

4. Run Unsloth's Flappy Bird test
5. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length (Gemma 3 supports 128K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
6. For conversation mode:

```bash
./llama.cpp/llama-mtmd-cli \
    --model unsloth/gemma-3-27b-it-GGUF/gemma-3-27b-it-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-3-27b-it-GGUF/mmproj-BF16.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 1.0 \
    --repeat-penalty 1.0 \
    --min-p 0.01 \
    --top-k 64 \
    --top-p 0.95
```

7. For non conversation mode to test Flappy Bird:

```bash
./llama.cpp/llama-cli \
    --model unsloth/gemma-3-27b-it-GGUF/gemma-3-27b-it-Q4_K_XL.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 1.0 \
    --repeat-penalty 1.0 \
    --min-p 0.01 \
    --top-k 64 \
    --top-p 0.95 \
    -no-cnv \
    --prompt "<start_of_turn>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<end_of_turn>\n<start_of_turn>model\n"
```

The full input from our <https://unsloth.ai/blog/deepseekr1-dynamic> 1.58bit blog is:

{% hint style="danger" %}
Remember to remove \<bos> since Gemma 3 auto adds a \<bos>!
{% endhint %}

{% code overflow="wrap" %}

```
<start_of_turn>user
Create a Flappy Bird game in Python. You must include these things:
1. You must use pygame.
2. The background color should be randomly chosen and is a light shade. Start with a light blue color.
3. Pressing SPACE multiple times will accelerate the bird.
4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.
5. Place on the bottom some land colored as dark brown or yellow chosen randomly.
6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.
7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.
8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.
The final game should be inside a markdown section in Python. Check your code for error
```

{% endcode %}

## :sloth: Fine-tuning Gemma 3 in Unsloth

**Unsloth is the only framework which works in float16 machines for Gemma 3 inference and training.** This means Colab Notebooks with free Tesla T4 GPUs also work!

* Try our new [Gemma 3 (270M) notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(270M\).ipynb) which makes the 270M parameter model very smart at playing chess and can predict the next chess move.
* Fine-tune Gemma 3 (4B) using our notebooks for: [**Text**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\).ipynb) or [**Vision**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision.ipynb)
* Or fine-tune [Gemma 3n (E4B)](/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune/gemma-3n-how-to-run-and-fine-tune) with [Text](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Conversational.ipynb) • [Vision](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Vision.ipynb) • [Audio](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Audio.ipynb)

{% hint style="warning" %}
When trying full fine-tune (FFT) Gemma 3, all layers default to float32 on float16 devices. Unsloth expects float16 and upcasts dynamically. To fix, run `model.to(torch.float16)` after loading, or use a GPU with bfloat16 support.
{% endhint %}

### Unsloth Fine-tuning Fixes

Our solution in Unsloth is 3 fold:

1. Keep all intermediate activations in bfloat16 format - can be float32, but this uses 2x more VRAM or RAM (via Unsloth's async gradient checkpointing)
2. Do all matrix multiplies in float16 with tensor cores, but manually upcasting / downcasting without the help of Pytorch's mixed precision autocast.
3. Upcast all other options that don't need matrix multiplies (layernorms) to float32.

## 🤔 Gemma 3 Fixes Analysis

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-6d6081cbb6f8b4b999c57803ac863c0ee9636ee1%2Foutput(1).png?alt=media" alt="" width="563"><figcaption><p>Gemma 3 1B to 27B exceed float16's maximum of 65504</p></figcaption></figure>

First, before we finetune or run Gemma 3, we found that when using float16 mixed precision, gradients and **activations become infinity** unfortunately. This happens in T4 GPUs, RTX 20x series and V100 GPUs where they only have float16 tensor cores.

For newer GPUs like RTX 30x or higher, A100s, H100s etc, these GPUs have bfloat16 tensor cores, so this problem does not happen! **But why?**

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e616649aa69253e34ea1d92b291dbda62d08044a%2Ffloat16%20bfloat16.png?alt=media" alt="" width="375"><figcaption><p>Wikipedia <a href="https://en.wikipedia.org/wiki/Bfloat16_floating-point_format">https://en.wikipedia.org/wiki/Bfloat16_floating-point_format</a></p></figcaption></figure>

Float16 can only represent numbers up to **65504**, whilst bfloat16 can represent huge numbers up to **10^38**! But notice both number formats use only 16bits! This is because float16 allocates more bits so it can represent smaller decimals better, whilst bfloat16 cannot represent fractions well.

But why float16? Let's just use float32! But unfortunately float32 in GPUs is very slow for matrix multiplications - sometimes 4 to 10x slower! So we cannot do this.


# Gemma 3n: How to Run & Fine-tune

Run Google's new Gemma 3n locally with Dynamic GGUFs on llama.cpp, Ollama, Open WebUI and fine-tune with Unsloth!

Google’s Gemma 3n multimodal model handles image, audio, video, and text inputs. Available in 2B and 4B sizes, it supports 140 languages for text and multimodal tasks. You can now run and fine-tune **Gemma-3n-E4B** and **E2B** locally using [Unsloth](https://github.com/unslothai/unsloth).

> **Fine-tune Gemma 3n with our** [**free Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Conversational.ipynb)

Gemma 3n has **32K context length**, 30s audio input, OCR, auto speech recognition (ASR), and speech translation via prompts.

<a href="#running-gemma-3n" class="button primary">Running Tutorial</a><a href="#fine-tuning-gemma-3n-with-unsloth" class="button secondary">Fine-tuning Tutorial</a><a href="#fixes-for-gemma-3n" class="button secondary">Fixes + Technical Analysis</a>

**Unsloth Gemma 3n (Instruct) uploads with optimal configs:**

<table><thead><tr><th width="249">Dynamic 2.0 GGUF (text only)</th><th width="285">Dynamic 4-bit Instruct (to fine-tune)</th><th>16-bit Instruct</th></tr></thead><tbody><tr><td><ul><li><a href="https://huggingface.co/unsloth/gemma-3n-E2B-it-GGUF">2B</a></li><li><a href="https://huggingface.co/unsloth/gemma-3n-E4B-it-GGUF">4B</a></li></ul></td><td><ul><li><a href="https://huggingface.co/unsloth/gemma-3n-E2B-it-unsloth-bnb-4bit">2B</a></li><li><a href="https://huggingface.co/unsloth/gemma-3n-E4B-it-unsloth-bnb-4bit">4B</a></li></ul></td><td><ul><li><a href="https://huggingface.co/unsloth/gemma-3n-E2B-it">2B</a></li><li><a href="https://huggingface.co/unsloth/gemma-3n-E4B-it">4B</a></li></ul></td></tr></tbody></table>

**See all our Gemma 3n uploads including base and more formats in** [**our collection here**](https://huggingface.co/collections/unsloth/gemma-3n-685d3874830e49e1c93f9339)**.**

## 🖥️ Running Gemma 3n

Currently Gemma 3n is only supported in **text format** for inference.

{% hint style="info" %}
We’ve [fixed issues](#fixes-for-gemma-3n) with GGUFs not working properly in Ollama only. Please redownload if using Ollama.
{% endhint %}

### :gear: Official Recommended Settings

According to the Gemma team, the official recommended settings for inference:

`temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0`

* Temperature of 1.0
* Top\_K of 64
* Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
* Top\_P of 0.95
* Repetition Penalty of 1.0. (1.0 means disabled in llama.cpp and transformers)
* Chat template:

  <pre data-overflow="wrap"><code><strong>&#x3C;bos>&#x3C;start_of_turn>user\nHello!&#x3C;end_of_turn>\n&#x3C;start_of_turn>model\nHey there!&#x3C;end_of_turn>\n&#x3C;start_of_turn>user\nWhat is 1+1?&#x3C;end_of_turn>\n&#x3C;start_of_turn>model\n
  </strong></code></pre>
* Chat template with `\n`newlines rendered (except for the last)

{% code overflow="wrap" %}

```
<bos><start_of_turn>user
Hello!<end_of_turn>
<start_of_turn>model
Hey there!<end_of_turn>
<start_of_turn>user
What is 1+1?<end_of_turn>
<start_of_turn>model\n
```

{% endcode %}

{% hint style="danger" %}
llama.cpp an other inference engines auto add a \<bos> - DO NOT add TWO \<bos> tokens! You should ignore the \<bos> when prompting the model!
{% endhint %}

### :llama: Tutorial: How to Run Gemma 3n in Ollama

{% hint style="success" %}
Please re download Gemma 3N quants or remove the old ones via Ollama since there are some bug fixes. You can do the below to delete the old file and refresh it:

```
ollama rm hf.co/unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL

ollama run hf.co/unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL
```

{% endhint %}

1. Install `ollama` if you haven't already!

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```bash
ollama run hf.co/unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL
```

### 📖 Tutorial: How to Run Gemma 3n in llama.cpp

{% hint style="info" %}
We would first like to thank [Xuan-Son Nguyen](https://x.com/ngxson) from Hugging Face, [Georgi Gerganov](https://x.com/ggerganov) from the llama.cpp team on making Gemma 3N work in llama.cpp!
{% endhint %}

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run`

```bash
./llama.cpp/llama-cli -hf unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL -ngl 99 --jinja
```

3. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision).

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/gemma-3n-E4B-it-GGUF",
    local_dir = "unsloth/gemma-3n-E4B-it-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*", "mmproj-BF16.gguf"], # For Q4_K_XL
)
```

4. Run the model.
5. Edit `--threads 32` for the number of CPU threads, `--ctx-size 32768` for context length (Gemma 3 supports 32K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
6. For conversation mode:

```bash
./llama.cpp/llama-cli \
    --model unsloth/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-3n-E4B-it-GGUF/mmproj-BF16.gguf \
    --ctx-size 32768 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 1.0 \
    --repeat-penalty 1.0 \
    --min-p 0.00 \
    --top-k 64 \
    --top-p 0.95
```

7. For non conversation mode to test Flappy Bird:

```bash
./llama.cpp/llama-cli \
    --model unsloth/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-3n-E4B-it-GGUF/mmproj-BF16.gguf \
    --ctx-size 32768 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 1.0 \
    --repeat-penalty 1.0 \
    --min-p 0.00 \
    --top-k 64 \
    --top-p 0.95 \
    -no-cnv \
    --prompt "<start_of_turn>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<end_of_turn>\n<start_of_turn>model\n"
```

{% hint style="danger" %}
Remember to remove \<bos> since Gemma 3N auto adds a \<bos>!
{% endhint %}

## 🦥 Fine-tuning Gemma 3n with Unsloth

Gemma 3n, like [Gemma 3](/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune#unsloth-fine-tuning-fixes-for-gemma-3), had issues running on <mark style="background-color:yellow;">**Flotat16 GPUs such as Tesla T4s in Colab**</mark>. You will encounter NaNs and infinities if you do not patch Gemma 3n for inference or finetuning. [More information below](#infinities-and-nan-gradients-and-activations).

* Fine-tune Gemma 3n-E4B with our [free Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Conversational.ipynb)
* **Audio:** Fine-tune Gemma 3n-E4B with our [**Audio only notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Audio.ipynb)
* **Vision**: Fine-tune Gemma 3n-E4B with our [**Vision only notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Vision.ipynb)

We also found that because Gemma 3n's unique architecture reuses hidden states in the vision encoder it poses another interesting quirk with [Gradient Checkpointing described below](#gradient-checkpointing-issues)

<mark style="background-color:purple;">**Unsloth is the only framework which works in float16 machines for Gemma 3n inference and training.**</mark> This means Colab Notebooks with free Tesla T4 GPUs also work! Overall, Unsloth makes Gemma 3n training 1.5x faster, 50% less VRAM and 4x longer context lengths.

Our free Gemma 3n Colab notebooks default to fine-tuning text layers. If you want to fine-tune vision or audio layers too, be aware this will require much more VRAM - beyond the 15GB free Colab or Kaggle provides. You *can* still fine-tune all layers including audio and vision and Unsloth also lets you fine-tune only specific areas, like just vision. Simply adjust as needed:

```python
model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers     = False, # False if not finetuning vision layers
    finetune_language_layers   = True,  # False if not finetuning language layers
    finetune_attention_modules = True,  # False if not finetuning attention layers
    finetune_mlp_modules       = True,  # False if not finetuning MLP layers
)
```

#### :trophy:Bonus Content

We also heard you guys wanted a <mark style="background-color:blue;">**Vision notebook for Gemma 3 (4B)**</mark> so here it is:

* Fine-tune Gemma 3 (4B) with Vision support using our [free Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_\(4B\)-Vision.ipynb)

{% hint style="info" %}
If you love Kaggle, Google is holding a competition where the best model fine-tuned with Gemma 3n and Unsloth will win a $10K prize! [See more here](https://www.kaggle.com/competitions/google-gemma-3n-hackathon).
{% endhint %}

## 🐛Fixes for Gemma 3n

### :sparkles:GGUF issues & fixes

Thanks to discussions from [Michael](https://github.com/mxyng) from the Ollama team and also [Xuan](https://x.com/ngxson) from Hugging Face, there were 2 issues we had to fix specifically for GGUFs:

1. The `add_shared_kv_layers` parameter was accidentally encoded in `float32` which is fine, but becomes slightly complicated to decode on Ollama's side - a simple change to `uint32` solves the issue. [Pull request](https://github.com/ggml-org/llama.cpp/pull/14450) addressing this issue.
2. The `per_layer_token_embd` layer should be Q8\_0 in precision. Anything lower does not function properly and errors out in the Ollama engine - to reduce issues for our community, we made this all Q8\_0 in all quants - unfortunately this does use more space.
   1. As an [update](https://huggingface.co/unsloth/gemma-3n-E4B-it-GGUF/discussions/4), [Matt](https://huggingface.co/WBB2500) mentioned we can also use Q4\_0, Q4\_1, Q5\_0, Q5\_1 for the embeddings - and we confirmed it does also work in Ollama! This means once again the smaller 2, 3 and 4bit quants are smaller in size, and don't need Q8\_0!

## :infinity:Infinities and NaN gradients and activations

{% columns %}
{% column %}
Gemma 3n just like Gemma 3 has issues on FP16 GPUs (e.g., Tesla T4s in Colab).

Our previous fixes for Gemma 3 is [discussed here](/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune). For Gemma 3, we found that activations exceed float16's maximum range of **65504.**

**Gemma 3N does not have this activation issue, but we still managed to encounter infinities!**
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-3f1aa0661c7919f8ad830fcbdf85a074d6a54bdf%2FGemma%203%20activation.webp?alt=media" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

To get to the bottom of these infinities, we plotted the absolute maximum weight entries for Gemma 3N, and we see the below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-e9b01a20a0a1cfcc41ef47cb29c5188d52d6a79d%2Foutput2.webp?alt=media" alt="" width="563"><figcaption></figcaption></figure>

We find that the green crosses are the Conv2D convolutional weights. We can see that the magnitude of Conv2D layers is much larger on average.

Below is a table for Conv2D weights which have large magnitudes. Our hypothesis is that during a Conv2D operation, large weights multiply and sum together, and **unfortunately by chance exceed float16's maximum range of 65504.** Bfloat16 is fine, since it's maximum range is 10^38.

| Name                                   | Max       |
| -------------------------------------- | --------- |
| msfa.ffn.pw\_proj.conv.weight          | 98.000000 |
| blocks.2.21.attn.key.down\_conv.weight | 37.000000 |
| blocks.2.32.pw\_exp.conv.weight        | 34.750000 |
| blocks.2.30.pw\_exp.conv.weight        | 33.750000 |
| blocks.2.34.pw\_exp.conv.weight        | 33.750000 |

### :sparkler:Solution to infinities

The naive solution is to `upcast` all Conv2D weights to float32 (if bfloat16 isn't available). But that would increase VRAM usage. To tackle this, we instead make use of `autocast` on the fly to upcast the weights and inputs to float32, and so we perform the accumulation in float32 as part of the matrix multiplication itself, without having to upcast the weights.

{% hint style="success" %}
Unsloth is the only framework that enables Gemma 3n inference and training on float16 GPUs, so Colab Notebooks with free Tesla T4s work!
{% endhint %}

### :checkered\_flag:Gradient Checkpointing issues

We found Gemma 3N's vision encoder to be quite unique as well since it re-uses hidden states. This unfortunately limits the usage of [Unsloth's gradient checkpointing](https://unsloth.ai/blog/long-context), which could have reduced VRAM usage significantly. since it cannot be applied to Vision encoder.

However, we still managed to leverage **Unsloth's automatic compiler** to optimize Gemma 3N!

### :cactus:Large losses during finetuning

We also found losses are interestingly very large during the start of finetuning - in the range of 6 to 7, but they do decrease over time quickly. We theorize this is either because of 2 possibilities:

1. There might be some implementation issue, but this is unlikely since inference seems to work.
2. <mark style="background-color:blue;">**Multi-modal models always seem to exhibit this behavior**</mark> - we found Llama 3.2 Vision's loss starts at 3 or 4, Pixtral at 8 or so, and Qwen 2.5 VL also 4 ish. Because Gemma 3N includes audio as well, it might amplify the starting loss. But this is just a hypothesis. We also found quantizing Qwen 2.5 VL 72B Instruct to have extremely high perplexity scores of around 30 or so, but the model interestingly performs fine.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-a37a8ce8ff2cfc3873a9f78acee3744c778692dc%2Foutput(3).png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="success" %}
**Fine-tune Gemma 3n with our** [**free Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Conversational.ipynb)
{% endhint %}

## 🛠️ Technical Analysis

### Gemma 3n : MatFormer

So what is so special about Gemma 3n you ask? It is based on [Matryoshka Transformer or MatFormer](https://arxiv.org/abs/2310.07707) architecture meaning that each transformer layer/block embeds/nests FFNs of progressively smaller sizes. Think of it like progressively smaller cups put inside one another. The training is done so that at inference time you can choose the size you want and get the most of the performance of the bigger models.

There is also Per Layer Embedding which can be cached to reduce memory usage at inference time. So the 2B model (E2B) is a sub-network inside the 4B (aka 5.44B) model that is achieved by both Per Layer Embedding caching and skipping audio and vision components focusing solely on text.

The MatFormer architecture, typically is trained with exponentially spaced sub-models aka of sizes `S`, `S/2, S/4, S/8` etc in each of the layers. So at training time, inputs are randomly forwarded through one of the said sub blocks giving every sub block equal chance to learn. Now the advantage is, at inference time, if you want the model to be 1/4th of the original size, you can pick `S/4` sized sub blocks in each layer.

You can also choose to **Mix and Match** where you pick say, `S/4` sized sub block of one layer, `S/2` sized sub block of another layer and `S/8` sized sub block of another layer. In fact, you can change the sub models you pick based on the input itself if you fancy so. Basically its like choose your own kind of structure at every layer. So by just training a model of one particular size, you are creating exponentially many models of smaller sizes. No learning goes waste. Pretty neat huh.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-0f2e2a7a9d652391166798dec37319252b399f8e%2Fimage.png?alt=media" alt="" width="563"><figcaption><p>Image from <a href="https://ai.google.dev/gemma/docs/gemma-3n">Gemma 3n model overview</a></p></figcaption></figure>

{% hint style="info" %}
**Fine-tune and try multimodal Gemma 3n inference with our** [**free Colab notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_\(4B\)-Conversational.ipynb)
{% endhint %}


# Qwen3-Coder: How to Run Locally

Run Qwen3-Coder-30B-A3B-Instruct and 480B-A35B locally with Unsloth Dynamic quants.

Qwen3-Coder is Qwen’s new series of coding agent models, available in 30B (**Qwen3-Coder-Flash**) and 480B parameters. **Qwen3-480B-A35B-Instruct** achieves SOTA coding performance rivalling Claude Sonnet-4, GPT-4.1, and [Kimi K2](/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally), with 61.8% on Aider Polygot and support for 256K (extendable to 1M) token context.

We also uploaded Qwen3-Coder with native <mark style="background-color:purple;">**1M context length**</mark> extended by YaRN and full-precision 8bit and 16bit versions. [Unsloth](https://github.com/unslothai/unsloth) also now supports fine-tuning and [RL](/docs/get-started/reinforcement-learning-rl-guide) of Qwen3-Coder.

{% hint style="success" %}
[**UPDATE:** We fixed tool-calling for Qwen3-Coder! ](#tool-calling-fixes)You can now use tool-calling seamlessly in llama.cpp, Ollama, LMStudio, Open WebUI, Jan etc. This issue was universal and affected all uploads (not just Unsloth), and we've communicated with the Qwen team about our fixes! [Read more](#tool-calling-fixes)
{% endhint %}

<a href="#run-qwen3-coder-30b-a3b-instruct" class="button primary">Run 30B-A3B</a><a href="#run-qwen3-coder-480b-a35b-instruct" class="button secondary">Run 480B-A35B</a>

{% hint style="success" %}
**Does** [**Unsloth Dynamic Quants**](/docs/basics/dynamic-3.0-ggufs) **work?** Yes, and very well. In third-party testing on the Aider Polyglot benchmark, the **UD-Q4\_K\_XL (276GB)** dynamic quant nearly matched the **full bf16 (960GB)** Qwen3-coder model, scoring 60.9% vs 61.8%. [More details here.](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/discussions/8)
{% endhint %}

#### **Qwen3 Coder - Unsloth Dynamic 2.0 GGUFs**:

| Dynamic 2.0 GGUF (to run)                                                                                                                                                                                                     | 1M Context Dynamic 2.0 GGUF                                                                                                                                                                                                         |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <ul><li><a href="https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF">30B-A3B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF">480B-A35B-Instruct</a></li></ul> | <ul><li><a href="https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-GGUF">30B-A3B-Instruct</a></li><li><a href="https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M-GGUF">480B-A35B-Instruct</a></li></ul> |

## 🖥️ **Running Qwen3-Coder**

Below are guides for the [**30B-A3B**](#run-qwen3-coder-30b-a3b-instruct) and [**480B-A35B**](#run-qwen3-coder-480b-a35b-instruct) variants of the model.

### :gear: Recommended Settings

Qwen recommends these inference settings for both models:

`temperature=0.7`, `top_p=0.8`, `top_k=20`, `repetition_penalty=1.05`

* <mark style="background-color:green;">**Temperature of 0.7**</mark>
* Top\_K of 20
* Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
* Top\_P of 0.8
* <mark style="background-color:green;">**Repetition Penalty of 1.05**</mark>
* Chat template:

  ```
  <|im_start|>user
  Hey there!<|im_end|>
  <|im_start|>assistant
  What is 1+1?<|im_end|>
  <|im_start|>user
  2<|im_end|>
  <|im_start|>assistant
  ```
* Recommended context output: 65,536 tokens (can be increased). Details here.

**Chat template/prompt format with newlines un-rendered**

{% code overflow="wrap" %}

```
<|im_start|>user\nHey there!<|im_end|>\n<|im_start|>assistant\nWhat is 1+1?<|im_end|>\n<|im_start|>user\n2<|im_end|>\n<|im_start|>assistant\n
```

{% endcode %}

<mark style="background-color:yellow;">**Chat template for tool calling**</mark> (Getting the current temperature for San Francisco). More details here for how to format tool calls.

```
<|im_start|>user
What's the temperature in San Francisco now? How about tomorrow?<|im_end|>
<|im_start|>assistant
<tool_call>\n<function=get_current_temperature>\n<parameter=location>\nSan Francisco, CA, USA
</parameter>\n</function>\n</tool_call><|im_end|>
<|im_start|>user
<tool_response>
{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}
</tool_response>\n<|im_end|>
```

{% hint style="info" %}
Reminder that this model supports only non-thinking mode and does not generate `<think></think>` blocks in its output. Meanwhile, specifying `enable_thinking=False` is no longer required.
{% endhint %}

### Run Qwen3-Coder-30B-A3B-Instruct:

To achieve inference speeds of 6+ tokens per second for our Dynamic 4-bit quant, have at least **18GB of unified memory** (combined VRAM and RAM) or **18GB of system RAM** alone. As a rule of thumb, your available memory should match or exceed the size of the model you’re using. E.g. the UD\_Q8\_K\_XL quant (full precision), which is 32.5GB, will require at least **33GB of unified memory** (VRAM + RAM) or **33GB of RAM** for optimal performance.

**NOTE:** The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds.

Given that this is a non thinking model, there is no need to set `thinking=False` and the model does not generate `<think> </think>` blocks.

{% hint style="info" %}
Follow the [**best practices above**](#recommended-settings). They're the same as the 480B model.
{% endhint %}

#### 🦙 Ollama: Run Qwen3-Coder-30B-A3B-Instruct Tutorial

1. Install `ollama` if you haven't already! You can only run models up to 32B in size.

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```bash
ollama run hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL
```

#### :sparkles: Llama.cpp: Run Qwen3-Coder-30B-A3B-Instruct Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. You can directly pull from HuggingFace via:

   ```bash
   ./llama.cpp/llama-cli \
       -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL \
       --jinja -ngl 99 --ctx-size 32768 \
       --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --repeat-penalty 1.05
   ```
3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD\_Q4\_K\_XL or other quantized versions. If downloads get stuck, see [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF",
    local_dir = "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

### Run Qwen3-Coder-480B-A35B-Instruct:

To achieve inference speeds of 6+ tokens per second for our 1-bit quant, we recommend at least **150GB of unified memory** (combined VRAM and RAM) or **150GB of system RAM** alone. As a rule of thumb, your available memory should match or exceed the size of the model you’re using. E.g. the Q2\_K\_XL quant, which is 180GB, will require at least **180GB of unified memory** (VRAM + RAM) or **180GB of RAM** for optimal performance.

**NOTE:** The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds.

{% hint style="info" %}
Follow the [**best practices above**](#recommended-settings). They're the same as the 30B model.
{% endhint %}

#### 📖 Llama.cpp: Run Qwen3-Coder-480B-A35B-Instruct Tutorial

For Coder-480B-A35B, we will specifically use Llama.cpp for optimized inference and a plethora of options.

{% hint style="success" %}
If you want a **full precision unquantized version**, use our `Q8_K_XL, Q8_0` or `BF16` versions!
{% endhint %}

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.

   ```bash
   apt-get update
   apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
   git clone https://github.com/ggml-org/llama.cpp
   cmake llama.cpp -B llama.cpp/build \
       -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
   cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
   cp llama.cpp/build/bin/llama-* llama.cpp
   ```
2. You can directly use llama.cpp to download the model but I normally suggest using `huggingface_hub` To use llama.cpp directly, do:

   ```bash
   ./llama.cpp/llama-cli \
       -hf unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF:Q2_K_XL \
       --ctx-size 16384 \
       --n-gpu-layers 99 \
       -ot ".ffn_.*_exps.=CPU" \
       --temp 0.7 \
       --min-p 0.0 \
       --top-p 0.8 \
       --top-k 20 \
       --repeat-penalty 1.05
   ```
3. Or, download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q2\_K\_XL, or other quantized versions..

   ```python
   # !pip install huggingface_hub hf_transfer
   import os
   os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
   from huggingface_hub import snapshot_download
   snapshot_download(
       repo_id = "unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF",
       local_dir = "unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF",
       allow_patterns = ["*UD-Q2_K_XL*"],
   )
   ```
4. Run the model in conversation mode and try any prompt.
5. Edit `--threads -1` for the number of CPU threads, `--ctx-size` 262114 for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% hint style="success" %}
Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity. More options discussed [here](#improving-generation-speed).
{% endhint %}

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/UD-Q2_K_XL/Qwen3-Coder-480B-A35B-Instruct-UD-Q2_K_XL-00001-of-00004.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --temp 0.7 \
    --min-p 0.0 \
    --top-p 0.8 \
    --top-k 20 \
    --repeat-penalty 1.05
```

{% endcode %}

{% hint style="success" %}
Also don't forget about the new Qwen3 update. Run [**Qwen3-235B-A22B-Instruct-2507**](/docs/models/tutorials/qwen3-next) locally with llama.cpp.
{% endhint %}

#### :tools: Improving generation speed

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.

The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel). You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster.

#### :triangular\_ruler:How to fit long context (256K to 1M)

To fit longer context, you can use <mark style="background-color:green;">**KV cache quantization**</mark> to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below.

`--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`

You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1`

You can also quantize the V cache, but you will need to <mark style="background-color:yellow;">**compile llama.cpp with Flash Attention**</mark> support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it.

We also uploaded 1 million context length GGUFs via YaRN scaling [here](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M-GGUF).

## :toolbox: Tool Calling Fixes

We managed to fix tool calling via `llama.cpp --jinja` specifically for serving through `llama-server`! If you’re downloading our 30B-A3B quants, no need to worry as these already include our fixes. For the 480B-A35B model, please:

1. Download the first file at <https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/tree/main/UD-Q2\\_K\\_XL> for UD-Q2\_K\_XL, and replace your current file
2. Use `snapshot_download` as usual as in <https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#llama.cpp-run-qwen3-tutorial> which will auto override the old files
3. Use the new chat template via `--chat-template-file`. See [GGUF chat template](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF?chat_template=default) or [chat\_template.jinja](https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct/raw/main/chat_template.jinja)
4. As an extra, we also made 1 single 150GB UD-IQ1\_M file (so Ollama works) at <https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/blob/main/Qwen3-Coder-480B-A35B-Instruct-UD-IQ1\\_M.gguf>

This should solve issues like: <https://github.com/ggml-org/llama.cpp/issues/14915>

### Using Tool Calling

To format the prompts for tool calling, let's showcase it with an example.

I created a Python function called `get_current_temperature` which is a function which should get the current temperature for a location. For now we created a placeholder function which will always return 21.6 degrees celsius. You should change this to a true function!!

{% code overflow="wrap" %}

```python
def get_current_temperature(location: str, unit: str = "celsius"):
    """Get current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format "City, State, Country".
        unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"])

    Returns:
        the temperature, the location, and the unit in a dict
    """
    return {
        "temperature": 26.1, # PRE_CONFIGURED -> you change this!
        "location": location,
        "unit": unit,
    }
```

{% endcode %}

Then use the tokenizer to create the entire prompt:

{% code overflow="wrap" %}

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3-Coder-480B-A35B-Instruct")

messages = [
    {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"},
    {'content': "", 'role': 'assistant', 'function_call': None, 'tool_calls': [
        {'id': 'ID', 'function': {'arguments': {"location": "San Francisco, CA, USA"}, 'name': 'get_current_temperature'}, 'type': 'function'},
    ]},
    {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool_call_id': 'ID'},
]

prompt = tokenizer.apply_chat_template(messages, tokenize = False)
```

{% endcode %}

## :bulb:Performance Benchmarks

{% hint style="info" %}
These official benchmarks are for the full BF16 checkpoint. To use this, simply use the `Q8_K_XL, Q8_0, BF16` checkpoints we uploaded - you can still use the tricks like MoE offloading for these versions as well!
{% endhint %}

Here are the benchmarks for the 480B model:

#### Agentic Coding

<table data-full-width="true"><thead><tr><th>Benchmark</th><th>Qwen3‑Coder 480B‑A35B‑Instruct</th><th>Kimi‑K2</th><th>DeepSeek‑V3-0324</th><th>Claude 4 Sonnet</th><th>GPT‑4.1</th></tr></thead><tbody><tr><td>Terminal‑Bench</td><td><strong>37.5</strong></td><td>30.0</td><td>2.5</td><td>35.5</td><td>25.3</td></tr><tr><td>SWE‑bench Verified w/ OpenHands (500 turns)</td><td><strong>69.6</strong></td><td>–</td><td>–</td><td>70.4</td><td>–</td></tr><tr><td>SWE‑bench Verified w/ OpenHands (100 turns)</td><td><strong>67.0</strong></td><td>65.4</td><td>38.8</td><td>68.0</td><td>48.6</td></tr><tr><td>SWE‑bench Verified w/ Private Scaffolding</td><td>–</td><td>65.8</td><td>–</td><td>72.7</td><td>63.8</td></tr><tr><td>SWE‑bench Live</td><td><strong>26.3</strong></td><td>22.3</td><td>13.0</td><td>27.7</td><td>–</td></tr><tr><td>SWE‑bench Multilingual</td><td><strong>54.7</strong></td><td>47.3</td><td>13.0</td><td>53.3</td><td>31.5</td></tr><tr><td>Multi‑SWE‑bench mini</td><td><strong>25.8</strong></td><td>19.8</td><td>7.5</td><td>24.8</td><td>–</td></tr><tr><td>Multi‑SWE‑bench flash</td><td><strong>27.0</strong></td><td>20.7</td><td>–</td><td>25.0</td><td>–</td></tr><tr><td>Aider‑Polyglot</td><td><strong>61.8</strong></td><td>60.0</td><td>56.9</td><td>56.4</td><td>52.4</td></tr><tr><td>Spider2</td><td><strong>31.1</strong></td><td>25.2</td><td>12.8</td><td>31.1</td><td>16.5</td></tr></tbody></table>

#### Agentic Browser Use

<table data-full-width="true"><thead><tr><th>Benchmark</th><th>Qwen3‑Coder 480B‑A35B‑Instruct</th><th>Kimi‑K2</th><th>DeepSeek‑V3 0324</th><th>Claude Sonnet‑4</th><th>GPT‑4.1</th></tr></thead><tbody><tr><td>WebArena</td><td><strong>49.9</strong></td><td>47.4</td><td>40.0</td><td>51.1</td><td>44.3</td></tr><tr><td>Mind2Web</td><td><strong>55.8</strong></td><td>42.7</td><td>36.0</td><td>47.4</td><td>49.6</td></tr></tbody></table>

#### Agentic Tool -Use

<table data-full-width="true"><thead><tr><th>Benchmark</th><th>Qwen3‑Coder 480B‑A35B‑Instruct</th><th>Kimi‑K2</th><th>DeepSeek‑V3 0324</th><th>Claude Sonnet‑4</th><th>GPT‑4.1</th></tr></thead><tbody><tr><td>BFCL‑v3</td><td><strong>68.7</strong></td><td>65.2</td><td>56.9</td><td>73.3</td><td>62.9</td></tr><tr><td>TAU‑Bench Retail</td><td><strong>77.5</strong></td><td>70.7</td><td>59.1</td><td>80.5</td><td>–</td></tr><tr><td>TAU‑Bench Airline</td><td><strong>60.0</strong></td><td>53.5</td><td>40.0</td><td>60.0</td><td>–</td></tr></tbody></table>


# MiniMax-M2.5: How to Run Guide

Run MiniMax-M2.5 locally on your own device!

MiniMax-M2.5 is a new open LLM achieving SOTA in coding, agentic tool use and search and office work, scoring 80.2% in [SWE-Bench](#benchmarks) Verified, 51.3% in Multi-SWE-Bench, and 76.3% in BrowseComp.

The **230B parameters** (10B active) model has a **200K context** window and unquantized bf16 requires **457GB**. Unsloth Dynamic **3-bit** GGUF reduces size to **101GB** **(-62%):** [**MiniMax-M2.5 GGUF**](https://huggingface.co/unsloth/MiniMax-M2.5-GGUF)

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA quantization performance - so 3-bit has important layers upcasted to 8 or 16-bit. You can also fine-tune the model via Unsloth, using multiGPUs.

{% hint style="success" %}
**Feb 26:** See how well our GGUF quants [perform on benchmarks here](#unsloth-gguf-benchmarks).
{% endhint %}

### :gear: Usage Guide

The 3-bit dynamic quant UD-Q3\_K\_XL uses **101GB** of disk space - this fits nicely on a **128GB unified memory Mac** for \~20+ tokens/s, and also works faster with a **1x16GB GPU and 96GB of RAM** for 25+ tokens/s. **2-bit** quants or the biggest 2-bit will fit on a 96GB device.

For near **full precision**, use `Q8_0` (8-bit) which utilizes 243GB and will fit on a 256GB RAM device / Mac for 10+ tokens/s.

{% hint style="success" %}
For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower.
{% endhint %}

### Recommended Settings

MiniMax recommends using the following parameters for best performance: `temperature=1.0`, `top_p = 0.95`, `top_k = 40`.

{% columns %}
{% column %}

| Default Settings (Most Tasks)      |
| ---------------------------------- |
| `temperature = 1.0`                |
| `top_p = 0.95`                     |
| `top_k = 40`                       |
| `repeat penalty = 1.0` or disabled |
| {% endcolumn %}                    |

{% column %}

* **Maximum context window:** `196,608`
* `Min_P = 0.01` (default might be 0.05)
* Default system prompt:

{% code overflow="wrap" %}

```
You are a helpful assistant. Your name is MiniMax-M2.5 and is built by MiniMax.
```

{% endcode %}
{% endcolumn %}
{% endcolumns %}

## Run MiniMax-M2.5 Tutorials:

For these tutorials, we will be utilizing the 3-bit [UD-Q3\_K\_XL](https://huggingface.co/unsloth/MiniMax-M2.5-GGUF?show_file_info=UD-Q3_K_XL%2FMiniMax-M2.5-UD-Q3_K_XL-00001-of-00004.gguf) quant which fits in a 128GB RAM device.

#### ✨ Run in llama.cpp

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:Q3\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 200K context length.

Follow this for **most default** use-cases:

```bash
export LLAMA_CACHE="unsloth/MiniMax-M2.5-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/MiniMax-M2.5-GGUF:UD-Q3_K_XL \
    --ctx-size 16384 \
    --flash-attn on \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-Q3_K_XL` (dynamic 4-bit quant) or other quantized versions like `UD-Q6_K_XL` . We recommend using our 4bit dynamic quant `UD-Q3_K_XL` to balance size and accuracy. If downloads get stuck, see [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

```bash
hf download unsloth/MiniMax-M2.5-GGUF \
    --local-dir unsloth/MiniMax-M2.5-GGUF \
    --include "*UD-Q3_K_XL*" # Use "*Q8_0*" for 8-bit
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/MiniMax-M2.5-GGUF/UD-Q3_K_XL/MiniMax-M2.5-UD-Q3_K_XL-00001-of-00004.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40 \
    --ctx-size 16384 \
    --seed 3407
```

{% endcode %}
{% endstep %}
{% endstepper %}

### 🦙 Llama-server & OpenAI's completion library

To deploy MiniMax-M2.5 for production, we use `llama-server` or OpenAI API. In a new terminal say via tmux, deploy the model via:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/MiniMax-M2.5-GGUF/UD-Q3_K_XL/MiniMax-M2.5-UD-Q3_K_XL-00001-of-00004.gguf \
    --alias "unsloth/MiniMax-M2.5" \
    --prio 3 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40 \
    --ctx-size 16384 \
    --port 8001
```

{% endcode %}

Then in a new terminal, after doing `pip install openai`, do:

{% code overflow="wrap" %}

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/MiniMax-M2.5",
    messages = [{"role": "user", "content": "Create a Snake game."},],
)
print(completion.choices[0].message.content)
```

{% endcode %}

## 📊 Benchmarks

### Unsloth GGUF Benchmarks

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhfO2gsbz2lWrZXg3ojyE%2FHCGBTzgboAASv_A.png?alt=media&amp;token=7d6334ca-4f3c-4946-aacd-d55527375fce" alt=""><figcaption></figcaption></figure>

[Benjamin Marie (third-party) benchmarked](https://x.com/bnjmn_marie/status/2027043753484021810/photo/1) **MiniMax-M2.5** using **Unsloth GGUF quantizations** on a **750-prompt mixed suite** (LiveCodeBench v6, MMLU Pro, GPQA, Math500), reporting both **overall accuracy** and **relative error increase** (how much more often the quantized model makes mistakes vs. the original).

Unsloth quants, no matter their precision perform much better than their non-Unsloth counterparts for both accuracy and relative error (despite being 8GB smaller).

**Key results:**

* **Best quality/size tradeoff here: `unsloth UD-Q4_K_XL`.**\
  It’s the closest to Original: only **6.0 points** down, and “only” **+22.8%** more errors than baseline.
* **Other Unsloth Q4 quants perform closely together (\~64.5–64.9 accuracy).**\
  `IQ4_NL`, `MXFP4_MOE`, and `UD-IQ2_XXS` are all basically the same quality on this benchmark, with **\~33–35%** more errors than Original.
* Unsloth GGUFs perform much better than other non-Unsloth GGUFs, e.g. see `lmstudio-community - Q4_K_M` (despite being 8GB smaller) and `AesSedai - IQ3_S`.

### Official Benchmarks

You can view further below for benchmarks in table format:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHtuBCFNe7qeVG538VVQV%2F97f76950-2c60-4a9b-bb96-228454afabe9.png?alt=media&amp;token=a16edc7e-db1a-4052-aa99-1b516539b896" alt="" width="563"><figcaption></figcaption></figure>

<table data-full-width="true"><thead><tr><th>Benchmark</th><th>MiniMax-M2.5</th><th>MiniMax-M2.1</th><th>Claude Opus 4.5</th><th>Claude Opus 4.6</th><th>Gemini 3 Pro</th><th>GPT-5.2 (thinking)</th></tr></thead><tbody><tr><td>AIME25</td><td>86.3</td><td>83.0</td><td>91.0</td><td>95.6</td><td>96.0</td><td>98.0</td></tr><tr><td>GPQA-D</td><td>85.2</td><td>83.0</td><td>87.0</td><td>90.0</td><td>91.0</td><td>90.0</td></tr><tr><td>SciCode</td><td>44.4</td><td>41.0</td><td>50.0</td><td>52.0</td><td>56.0</td><td>52.0</td></tr><tr><td>IFBench</td><td>70.0</td><td>70.0</td><td>58.0</td><td>53.0</td><td>70.0</td><td>75.0</td></tr><tr><td>AA-LCR</td><td>69.5</td><td>62.0</td><td>74.0</td><td>71.0</td><td>71.0</td><td>73.0</td></tr><tr><td>SWE-Bench Verified</td><td>80.2</td><td>74.0</td><td>80.9</td><td>80.8</td><td>78.0</td><td>80.0</td></tr><tr><td>SWE-Bench Pro</td><td>55.4</td><td>49.7</td><td>56.9</td><td>55.4</td><td>54.1</td><td>55.6</td></tr><tr><td>Terminal Bench 2</td><td>51.7</td><td>47.9</td><td>53.4</td><td>55.1</td><td>54.0</td><td>54.0</td></tr><tr><td>HLE w/o tools</td><td>19.4</td><td>22.2</td><td>28.4</td><td>30.7</td><td>37.2</td><td>31.4</td></tr><tr><td>Multi-SWE-Bench</td><td>51.3</td><td>47.2</td><td>50.0</td><td>50.3</td><td>42.7</td><td>—</td></tr><tr><td>SWE-Bench Multilingual</td><td>74.1</td><td>71.9</td><td>77.5</td><td>77.8</td><td>65.0</td><td>72.0</td></tr><tr><td>VIBE-Pro (AVG)</td><td>54.2</td><td>42.4</td><td>55.2</td><td>55.6</td><td>36.9</td><td>—</td></tr><tr><td>BrowseComp (w/ctx)</td><td>76.3</td><td>62.0</td><td>67.8</td><td>84.0</td><td>59.2</td><td>65.8</td></tr><tr><td>Wide Search</td><td>70.3</td><td>63.2</td><td>76.2</td><td>79.4</td><td>57.0</td><td>—</td></tr><tr><td>RISE</td><td>50.2</td><td>34.0</td><td>50.5</td><td>62.5</td><td>36.8</td><td>50.0</td></tr><tr><td>BFCL multi-turn</td><td>76.8</td><td>37.4</td><td>68.0</td><td>63.3</td><td>61.0</td><td>—</td></tr><tr><td>τ² Telecom</td><td>97.8</td><td>87.0</td><td>98.2</td><td>99.3</td><td>98.0</td><td>98.7</td></tr><tr><td>MEWC</td><td>74.4</td><td>55.6</td><td>82.1</td><td>89.8</td><td>78.7</td><td>41.3</td></tr><tr><td>GDPval-MM</td><td>59.0</td><td>24.6</td><td>61.1</td><td>73.5</td><td>28.1</td><td>54.5</td></tr><tr><td>Finance Modeling</td><td>21.6</td><td>17.3</td><td>30.1</td><td>33.2</td><td>15.0</td><td>20.0</td></tr></tbody></table>

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FfHyJbhPs4k10iiauD3zo%2F1f5a4e78-1a5c-4263-8a65-36c6fe703041.png?alt=media&amp;token=346519a3-5dee-4ea1-b395-c7de12b6f6cd" alt="" width="563"><figcaption><p>Coding Core Benchmark Scores</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhJmbCVK6CNZPw2FSJLPT%2F2003295c-001c-4381-af89-8859c197b5a0.png?alt=media&amp;token=aaa9968d-f694-4b1e-b734-086c2cf8988b" alt="" width="563"><figcaption><p>Search and Tool Use</p></figcaption></figure></div>

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FaG1VqXpVwoCWG3RxMWDL%2F91c4825c-1813-4cad-9e36-4b69c6cd0272.png?alt=media&amp;token=ae956a3f-a25e-4ebc-80cb-97733573e654" alt=""><figcaption><p>Tasks Completed per 100</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fj2PwoDid8zWRyYnXA6of%2F8c25f392-275d-4730-aa92-e9ea27315d83.png?alt=media&amp;token=a59cf4d3-ff63-4383-90c8-2d089db19b50" alt=""><figcaption><p>Office Capabilities</p></figcaption></figure></div>


# DeepSeek-OCR 2: How to Run & Fine-tune Guide

Guide on how to run and fine-tune DeepSeek-OCR-2 locally.

**DeepSeek-OCR 2** is the new 3B-parameter model for SOTA vision and document understanding released on Jan 27, 2026 by DeepSeek. The model focuses on image-to-text with stronger visual reasoning, not just text extraction.

DeepSeek-OCR 2 introduces DeepEncoder V2, which enables the model to 'see' an image in the same logical order as a human.

Unlike traditional vision LLMs that scan images in a fixed grid (top-left → bottom-right), DeepEncoder V2 builds a global understanding first, then learns a human-like reading order, what to attend to first, next, and so on. This boosts OCR on complex layouts by better following columns, linking labels to values, reading tables coherently, and handling mixed text + structure.

You can now fine-tune DeepSeek-OCR 2 in Unsloth via our [**free fine-tuning notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_\(3B\).ipynb)**.** We demonstrated a [88.6% improvement](#fine-tuning-deepseek-ocr) for language understanding.

<a href="/docs/models/tutorials/deepseek-ocr-2#running-deepseek-ocr-2" class="button primary">Running DeepSeek-OCR 2</a><a href="/docs/models/tutorials/deepseek-ocr-2#fine-tuning-deepseek-ocr-2" class="button secondary">Fine-tuning DeepSeek-OCR 2</a>

## 🖥️ **Running DeepSeek-OCR 2**

In order to run the model, like the first model, DeepSeek-OCR 2 was edited to enable inference & training on the latest transformers (no accuracy change). You can find [it here](https://huggingface.co/unsloth/DeepSeek-OCR-2).

To run the model in [transformers](#transformers-run-deepseek-ocr-2-tutorial) or [Unsloth](#unsloth-run-deepseek-ocr-tutorial), here are the recommended settings:

### :gear: Recommended Settings

DeepSeek recommends these settings:

* <mark style="background-color:blue;">**Temperature = 0.0**</mark>
* `max_tokens = 8192`
* `ngram_size = 30`
* `window_size = 90`

**Support Modes - Dynamic resolution:**

* Default: (0-6)×768×768 + 1×1024×1024 — (0-6)×144 + 256 visual tokens

**Prompts examples:**

```
# document: <image>\n<|grounding|>Convert the document to markdown.
# other image: <image>\n<|grounding|>OCR this image.
# without layouts: <image>\nFree OCR.
# figures in document: <image>\nParse the figure.
# general: <image>\nDescribe this image in detail.
# rec: <image>\nLocate <|ref|>xxxx<|/ref|> in the image.
```

<div align="center"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FGff3UBRgiu0aFXsbwll2%2Ffig1.png?alt=media&amp;token=58864777-114a-450a-ab56-3cabc9219cf7" alt="" width="375"><figcaption><p>Turns any document into markdown using Visual Causal Flow.</p></figcaption></figure></div>

### 🦥 Unsloth: Run DeepSeek-OCR 2 Tutorial

1. Obtain the latest `unsloth` via `pip install --upgrade unsloth` . If you already have Unsloth, update it via `pip install --upgrade --force-reinstall --no-deps --no-cache-dir unsloth unsloth_zoo`
2. Then use the code below to run DeepSeek-OCR 2:

{% code overflow="wrap" %}

```python
from unsloth import FastVisionModel
import torch
from transformers import AutoModel
import os
os.environ["UNSLOTH_WARN_UNINITIALIZED"] = '0'

from huggingface_hub import snapshot_download
snapshot_download("unsloth/DeepSeek-OCR-2", local_dir = "deepseek_ocr")
model, tokenizer = FastVisionModel.from_pretrained(
    "./deepseek_ocr",
    load_in_4bit = False, # Use 4bit to reduce memory use. False for 16bit LoRA.
    auto_model = AutoModel,
    trust_remote_code = True,
    unsloth_force_compile = True,
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
)

prompt = "<image>\nFree OCR. "
image_file = 'your_image.jpg'
output_path = 'your/output/dir'
res = model.infer(tokenizer, prompt=prompt, image_file=image_file, output_path = output_path, base_size = 1024, image_size = 640, crop_mode=True, save_results = True, test_compress = False)
```

{% endcode %}

### 🤗 Transformers: Run DeepSeek-OCR 2 Tutorial

Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.12.9 + CUDA11.8:

```bash
torch==2.6.0
transformers==4.46.3
tokenizers==0.20.3
einops
addict 
easydict
pip install flash-attn==2.7.3 --no-build-isolation
```

```python
from transformers import AutoModel, AutoTokenizer
import torch
import os
os.environ["CUDA_VISIBLE_DEVICES"] = '0'
model_name = 'unsloth/DeepSeek-OCR-2'

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, _attn_implementation='flash_attention_2', trust_remote_code=True, use_safetensors=True)
model = model.eval().cuda().to(torch.bfloat16)

# prompt = "<image>\nFree OCR. "
prompt = "<image>\n<|grounding|>Convert the document to markdown. "
image_file = 'your_image.jpg'
output_path = 'your/output/dir'

res = model.infer(tokenizer, prompt=prompt, image_file=image_file, output_path = output_path, base_size = 1024, image_size = 768, crop_mode=True, save_results = True)
```

## 🦥 **Fine-tuning DeepSeek-OCR 2**

Unsloth now supports fine-tuning of DeepSeek-OCR 2. Like the first model, you'll need to use our [custom upload](https://huggingface.co/unsloth/DeepSeek-OCR-2) for it to work on `transformers` (no accuracy change). Like the first model, Unsloth trains DeepSeek-OCR-2 1.4x faster with 40% less VRAM and 5x longer context lengths with no accuracy degradation.\
\
You can now fine-tune DeepSeek-OCR 2 via our free Colab notebook.

* DeepSeek-OCR 2: [Fine-tuning only notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_2_\(3B\).ipynb)

See below for CER (Character Error Rate) accuracy improvements on the Persian language:

#### Per-sample CER (10 samples)

| idx  | OCR1 before | OCR1 after | OCR2 before | OCR2 after |
| ---- | ----------: | ---------: | ----------: | ---------: |
| 1520 |      1.0000 |     0.8000 |     10.4000 |     1.0000 |
| 1521 |      0.0000 |     0.0000 |      2.6809 |     0.0213 |
| 1522 |      2.0833 |     0.5833 |      4.4167 |     1.0000 |
| 1523 |      0.2258 |     0.0645 |      0.8710 |     0.0968 |
| 1524 |      0.0882 |     0.1176 |      2.7647 |     0.0882 |
| 1525 |      0.1111 |     0.1111 |      0.9444 |     0.2222 |
| 1526 |      2.8571 |     0.8571 |      4.2857 |     0.7143 |
| 1527 |      3.5000 |     1.5000 |     13.2500 |     1.0000 |
| 1528 |      2.7500 |     1.5000 |      1.0000 |     1.0000 |
| 1529 |      2.2500 |     0.8750 |      1.2500 |     0.8750 |

#### Average CER (10 samples)

* **OCR1:** before **1.4866**, after **0.6409** (**-57%**)
* **OCR2:** before **4.1863**, after **0.6018** (**-86%**)

## 📊 Benchmarks

Benchmarks for DeepSeek-OCR 2 model are derived from the official research paper.

**Table 1:** Comprehensive evaluation of document reading on OmniDocBench v1.5. V-token𝑚𝑎𝑥\
represents the maximum number of visual tokens used per page in this benchmark. R-order\
denotes reading order. Except for DeepSeek OCR and DeepSeek OCR 2, all other model results\
in this table are sourced from the OmniDocBench repository.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F7CjBxsi10P3kqyF3utpq%2FScreenshot%202026-01-27%20at%201.14.02%E2%80%AFAM.png?alt=media&amp;token=08fc9963-15d1-4d7a-9fb5-93749913928c" alt="" width="375"><figcaption></figcaption></figure>

**Table 2:** Edit Distances for different categories of document-elements in OmniDocBench v1.5.\
V-token𝑚𝑎𝑥 denotes the lowest maximum number of visual tokens.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FcTVu52ugTwQAqLdnQLLz%2FScreenshot%202026-01-27%20at%201.14.15%E2%80%AFAM.png?alt=media&amp;token=ec92a4a1-facb-4b63-90cd-73d23a41dcfb" alt="" width="563"><figcaption><p>Outperforms Gemini-3 Pro on the OmniDocBench</p></figcaption></figure>


# GLM-4.7: How to Run Locally Guide

A guide on how to run Z.ai GLM-4.7 model on your own local device!

GLM-4.7 is Z.ai’s latest thinking model, delivering stronger coding, agent, and chat performance than [GLM-4.6](/docs/models/tutorials/glm-4.6-how-to-run-locally). It achieves SOTA performance on on SWE-bench (73.8%, +5.8), SWE-bench Multilingual (66.7%, +12.9), and Terminal Bench 2.0 (41.0%, +16.5).

The full 355B parameter model requires **400GB** of disk space, while the Unsloth Dynamic 2-bit GGUF reduces the size to **134GB** (-**75%)**. [**GLM-4.7-GGUF**](https://huggingface.co/unsloth/GLM-4.7-GGUF)

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and Aider performance, meaning you can run & fine-tune quantized GLM LLMs with minimal accuracy loss.

### :gear: Usage Guide

The 2-bit dynamic quant UD-Q2\_K\_XL uses 135GB of disk space - this works well in a **1x24GB card and 128GB of RAM** with MoE offloading. The 1-bit UD-TQ1 GGUF also **works natively in Ollama**!

{% hint style="info" %}
You must use `--jinja` for llama.cpp quants - this uses our [fixed chat templates](#chat-template-bug-fixes) and enables the correct template! You might get incorrect results if you do not use `--jinja`
{% endhint %}

The 4-bit quants will fit in a 1x 40GB GPU (with MoE layers offloaded to RAM). Expect around 5 tokens/s with this setup if you have bonus 165GB RAM as well. It is recommended to have at least 205GB RAM to run this 4-bit. For optimal performance you will need at least 205GB unified memory or 205GB combined RAM+VRAM for 5+ tokens/s. To learn how to increase generation speed and fit longer contexts, [read here](#improving-generation-speed).

{% hint style="success" %}
Though not a must, for best performance, have your VRAM + RAM combined equal to the size of the quant you're downloading. If not, hard drive / SSD offloading will work with llama.cpp, just inference will be slower. Also use `--fit on` in `llama.cpp` to auto enable maximum GPU usage!
{% endhint %}

### Recommended Settings

Use distinct settings for different use cases. Recommended settings for default and multi-turn agentic use cases:

| Default Settings (Most Tasks)                                      | Terminal Bench, SWE Bench Verified                                 |
| ------------------------------------------------------------------ | ------------------------------------------------------------------ |
| <mark style="background-color:green;">**temperature = 1.0**</mark> | <mark style="background-color:green;">**temperature = 0.7**</mark> |
| <mark style="background-color:green;">**top\_p = 0.95**</mark>     | <mark style="background-color:green;">**top\_p = 1.0**</mark>      |
| `131072` **max new tokens**                                        | `16384` **max new tokens**                                         |

* Use `--jinja` for llama.cpp variants - we **fixed some chat template issues as well!**
* **Maximum context window:** `131,072`

## Run GLM-4.7 Tutorials:

See our step-by-step guides for running GLM-4.7 in [Ollama](#run-in-ollama) and [llama.cpp](#run-in-llama.cpp).

### ✨ Run in llama.cpp

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:Q2\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 128K context length.

```bash
export LLAMA_CACHE="unsloth/GLM-4.7-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/GLM-4.7-GGUF:UD-Q2_K_XL \
    --jinja \
    --ctx-size 16384 \
    --flash-attn on \
    --temp 1.0 \
    --top-p 0.95 \
    --fit on
```

{% hint style="info" %}
**Use `--fit on` introduced 15th Dec 2025 for maximum usage of your GPU and CPU.**

Optionally, try `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}
{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-`Q2\_K\_XL (dynamic 2bit quant) or other quantized versions like `Q4_K_XL` . We <mark style="background-color:green;">**recommend using our 2.7bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>.

```bash
pip install -U huggingface_hub
hf download unsloth/GLM-4.7-GGUF \
    --local-dir unsloth/GLM-4.7-GGUF \
    --include "*UD-Q2_K_XL*" # Use "*UD-TQ1_0*" for Dynamic 1bit
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/GLM-4.7-GGUF/UD-Q2_K_XL/GLM-4.7-UD-Q2_K_XL-00001-of-00003.gguf \
    --jinja \
    --temp 1.0 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --seed 3407 \
    --fit on
```

{% endcode %}
{% endstep %}
{% endstepper %}

### :llama: Run in Ollama

{% stepper %}
{% step %}
Install `ollama` if you haven't already! To run more variants of the model, [see here](https://unsloth.ai/docs/models/tutorials/pages/eK0BfjMHNrfvfe4HI6Pk#run-in-llama.cpp).

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

{% endstep %}

{% step %}
Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```bash
OLLAMA_MODELS=unsloth ollama serve &

OLLAMA_MODELS=unsloth ollama run hf.co/unsloth/GLM-4.7-GGUF:TQ1_0
```

{% endstep %}

{% step %}
To run other quants, you need to first merge the GGUF split files into 1 like the code below. Then you will need to run the model locally.

```bash
./llama.cpp/llama-gguf-split --merge \
  GLM-4.7-GGUF/GLM-4.7-UD-Q2_K_XL/GLM-4.7-UD-Q2_K_XL-00001-of-00003.gguf \
	merged_file.gguf
```

```bash
OLLAMA_MODELS=unsloth ollama serve &

OLLAMA_MODELS=unsloth ollama run merged_file.gguf
```

{% endstep %}
{% endstepper %}

### ✨ Deploy with llama-server and OpenAI's completion library

To use llama-server for deployment, use the following command:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/GLM-4.7-GGUF/UD-Q2_K_XL/GLM-4.7-UD-Q2_K_XL-00001-of-00003.gguf \
    --alias "unsloth/GLM-4.7" \
    --fit on \
    --prio 3 \
    --temp 1.0 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

{% endcode %}

Then use OpenAI's Python library after `pip install openai` :

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/GLM-4.7",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
```

### :hammer:Tool Calling with GLM 4.7

See [Tool Calling Guide](/docs/basics/tool-calling-guide-for-local-llms) for more details on how to do tool calling. In a new terminal (if using tmux, use CTRL+B+D), we create some tools like adding 2 numbers, executing Python code, executing Linux functions and much more:

{% code expandable="true" %}

```python
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def subtract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([
        "A long time ago in a galaxy far far away...",
        "There were 2 friends who loved sloths and code...",
        "The world was ending because every sloth evolved to have superhuman intelligence...",
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "subtract_number": subtract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [
    {
        "type": "function",
        "function": {
            "name": "add_number",
            "description": "Add two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "multiply_number",
            "description": "Multiply two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "subtract_number",
            "description": "Subtract two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "write_a_story",
            "description": "Writes a random story.",
            "parameters": {
                "type": "object",
                "properties": {},
                "required": [],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "terminal",
            "description": "Perform operations from the terminal.",
            "parameters": {
                "type": "object",
                "properties": {
                    "command": {
                        "type": "string",
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",
                    },
                },
                "required": ["command"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "python",
            "description": "Call a Python interpreter with some Python code that will be ran.",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {
                        "type": "string",
                        "description": "The Python code to run",
                    },
                },
                "required": ["code"],
            },
        },
    },
]
```

{% endcode %}

We then use the below functions (copy and paste and execute) which will parse the function calls automatically and call the OpenAI endpoint for any model:

{% code overflow="wrap" expandable="true" %}

```python
from openai import OpenAI
def unsloth_inference(
    messages,
    temperature = 0.7,
    top_p = 0.95,
    top_k = 40,
    min_p = 0.01,
    repetition_penalty = 1.0,
):
    messages = messages.copy()
    openai_client = OpenAI(
        base_url = "http://127.0.0.1:8001/v1",
        api_key = "sk-no-key-required",
    )
    model_name = next(iter(openai_client.models.list())).id
    print(f"Using model = {model_name}")
    has_tool_calls = True
    original_messages_len = len(messages)
    while has_tool_calls:
        print(f"Current messages = {messages}")
        response = openai_client.chat.completions.create(
            model = model_name,
            messages = messages,
            temperature = temperature,
            top_p = top_p,
            tools = tools if tools else None,
            tool_choice = "auto" if tools else None,
            extra_body = {"top_k": top_k, "min_p": min_p, "repetition_penalty" :repetition_penalty,}
        )
        tool_calls = response.choices[0].message.tool_calls or []
        content = response.choices[0].message.content or ""
        tool_calls_dict = [tc.to_dict() for tc in tool_calls] if tool_calls else tool_calls
        messages.append({"role": "assistant", "tool_calls": tool_calls_dict, "content": content,})
        for tool_call in tool_calls:
            fx, args, _id = tool_call.function.name, tool_call.function.arguments, tool_call.id
            out = MAP_FN[fx](**json.loads(args))
            messages.append({"role": "tool", "tool_call_id": _id, "name": fx, "content": str(out),})
        else:
            has_tool_calls = False
    return messages
```

{% endcode %}

After launching GLM 4.7 via `llama-server` like in [#deploy-with-llama-server-and-openais-completion-library](#deploy-with-llama-server-and-openais-completion-library "mention") or see [Tool Calling Guide](/docs/basics/tool-calling-guide-for-local-llms) for more details, we then can do some tool calls:

**Tool Call for mathematical operations for GLM 4.7**

{% code overflow="wrap" %}

```python
messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "What is today's date plus 3 days?"}],
}]
unsloth_inference(messages, temperature = 0.7, top_p = 1.0, top_k = -1, min_p = 0.00)
```

{% endcode %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FoFkZ20QOSGdzT4iz2SOB%2Fimage.png?alt=media&amp;token=e4ca30b0-dcec-4a26-b019-dd33f0600949" alt=""><figcaption></figcaption></figure>

**Tool Call to execute generated Python code for GLM 4.7**

{% code overflow="wrap" %}

```python
messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "Create a Fibonacci function in Python and find fib(20)."}],
}]
unsloth_inference(messages, temperature = 0.7, top_p = 1.0, top_k = -1, min_p = 0.00)
```

{% endcode %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhS8sWtZwjwerElezCc2C%2Fimage.png?alt=media&amp;token=39032ef8-386e-4837-8dd2-c552c80a3ee3" alt=""><figcaption></figcaption></figure>

### :snowboarder: Improving generation speed

**Use `--fit on` introduced 15th Dec 2025 for maximum usage of your GPU and CPU. See** [**https://github.com/ggml-org/llama.cpp/pull/16653**](https://github.com/ggml-org/llama.cpp/pull/16653) **`--fit on` auto offloads as much of the model as possible to the GPU, then places the rest on CPU.**

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.

Llama.cpp also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel). You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster.

### 📐How to fit long context (full 128K)

To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below.

`--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`

You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1`

You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. Then you can use together with `--cache-type-k` :

`--cache-type-v f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`


# How to Run Qwen-Image-2512 Locally in ComfyUI

Step-by-step tutorial for running Qwen-Image-2512 on your local device with ComfyUI.

**Qwen-Image-2512** is the December update to Qwen's text-to-image foundational models. The model is the top performing open-source diffusion model and this guide will teach you how to run it locally via [Unsloth](https://github.com/unslothai/unsloth) GGUF and ComfyUI.

Qwen-Image-2512 features: more realistic looking people; richer details in landscapes/textures; and more accurate text rendering. **Uploads:** [GGUF](https://huggingface.co/unsloth/Qwen-Image-2512-GGUF) • [FP8](https://huggingface.co/unsloth/Qwen-Image-2512-FP8) • [4-bit BnB](https://huggingface.co/unsloth/Qwen-Image-2512-unsloth-bnb-4bit)

The quants use [Unsloth Dynamic](/docs/basics/dynamic-3.0-ggufs) methodology which upcasts important layers to higher precision to recover more accuracy. Thank you Qwen for allowing Unsloth day 0 support.

## 📖 ComfyUI Tutorial

To run, you don't need a GPU, just a CPU with RAM will work. For best results, ensure your total usable memory (RAM + VRAM / unified) is larger than the GGUF size; e.g. 4-bit (Q4\_K\_M) `unsloth/Qwen-Image-Edit-2512-GGUF` is 13.1 GB, so you should have 13.2+ GB of combined memory.

[ComfyUI](https://github.com/Comfy-Org/ComfyUI) is an open-source diffusion model GUI, API, and backend that uses a node-based (graph/flowchart) interface. This guide will focus on machines with CUDA, but instructions to build with on Apple or CPU are similar.

### #1. Install & Setup

To install ComfyUI, you can download the desktop app on Windows or Mac devices [here](https://www.comfy.org/download). Otherwise, to setup ComfyUI for running GGUF models run the following:

```bash
mkdir comfy_ggufs
cd comfy_ggufs
python -m venv .venv
source .venv/bin/activate

git clone https://github.com/Comfy-Org/ComfyUI.git
cd ComfyUI
pip install -r requirements.txt

cd custom_nodes
git clone https://github.com/city96/ComfyUI-GGUF
cd ComfyUI-GGUF
pip install -r requirements.txt
cd ../..
```

### #2. Download Models

Diffusion models typically need 3 models. A Variational AutoEncoder (VAE) that encodes image pixel space to latent space, a text encoder to translate text to input embeddings, and the actual diffusion transformer. You can find all Unsloth diffusion GGUFs in our [Collection here](https://huggingface.co/collections/unsloth/unsloth-diffusion-ggufs).

Both the diffusion model and text encoder can be GGUF format while we typically use safetensors for the vae. According to [Qwen's repo](https://huggingface.co/Qwen/Qwen-Image-2512/blob/main/text_encoder/config.json), we shall use Qwen2.5-VL and not [Qwen3-VL](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune). Let's download the models we will use (Note: You can also also use our [FP8 upload](https://huggingface.co/unsloth/Qwen-Image-2512-FP8) in ComfyUI):

```bash
cd models

## Diffusion Models
curl -L -C - -o unet/qwen-image-2512-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen-Image-2512-GGUF/resolve/main/qwen-image-2512-Q4_K_M.gguf
curl -L -C - -o unet/qwen-image-edit-2511-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF/resolve/main/qwen-image-edit-2511-Q4_K_M.gguf
 
## Text Encoder + Vision Tower + VAE   
curl -L -C - -o text_encoders/Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf \
  https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF/resolve/main/Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf
curl -L -C - -o text_encoders/Qwen2.5-VL-7B-Instruct-mmproj-BF16.gguf \
  https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF/resolve/main/mmproj-BF16.gguf
curl -L -C - -o vae/qwen_image_vae.safetensors \
  https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/vae/qwen_image_vae.safetensors
```

See GGUF uploads for: [Qwen-Image-2512](https://huggingface.co/unsloth/Qwen-Image-2512-GGUF), [Qwen-Image-Edit-2511](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF), and [Qwen-Image-Layered](https://huggingface.co/unsloth/Qwen-Image-Layered-GGUF)

{% hint style="warning" %}
The format of the vae and diffusion model might be different than the diffusers checkpoints if using checkpoints other than the ones above. Only use checkpoints that are compatible with ComfyUI.
{% endhint %}

These files must be in the correct folders for ComfyUI to see them. In addition the vision tower stored in the mmproj file must use the same prefix as the text encoder.

Download reference images to be used later as well:

```bash
curl -L -C - -o ../input/sloth1.jpg \
    "https://unsloth.ai/cgi/image/_1d5a5685-2d88-44ca-b50f-ba432cd646ef_9CGCY8lvw4D9JkOdueqsk.jpeg?width=1920&quality=80&format=jpeg"

curl -L -C - -o ../input/sloth2.jpg \
    "https://unsloth.ai/cgi/image/UnSloth_GPU_Front_-_Confetti_ArcSk-MR4MMN215UutOFZ.png?width=1920&quality=80&format=jpeg"
```

### #3. Workflow and Hyperparameters

For more info you can also view our detailed [Run GGUFs in ComfyUI](/docs/blog/comfyui#workflow-and-hyperparameters-1) Guide.

Navigate to the main ComfyUI directory and run:

```bash
python main.py
```

{% hint style="info" %}
`python main.py --cpu` to run with CPU, but will be slow.
{% endhint %}

This will launch a web server that allows you to access `https://127.0.0.1:8188` . If you are running this on the cloud, you'll need to make sure port forwarding is setup to access on your local machine.

Workflows are saved as JSON files embedded in output images (PNG metadata) or as separate `.json` files. You can:

* Drag & drop an image into ComfyUI to load its workflow
* Export/import workflows via the menu
* Share workflows as JSON files

Below are two examples of Qwen-Image-2512 and Qwen-Image-Edit-2511 json files which you can download and use:

{% file src="/files/GEtfTCPAWM4rFYgNvYfd" %}

For our workflow, we default to **1024×1024** as a practical middle ground. While the model supports native resolution (1328×1328), generating at native typically increases runtime by **\~50%**. Since GGUF adds overhead and 40 steps is a relatively long run, 1024×1024 keeps generation time reasonable. If needed, you can increase resolution to 1328.

{% hint style="warning" %}
For more realistic results, skip keywords like “photorealistic” or “digital rendering” or “3d render” and use terms like “photograph” instead.
{% endhint %}

{% hint style="info" %}
For negative prompts, it’s best to use an NLP-style approach: describe in **natural language** what you *don’t* want in the image. Packing in too many keywords can hurt results instead of making it more specific.
{% endhint %}

{% file src="/files/Z0JySsOLKNNUZb6tzGDk" %}

{% columns %}
{% column %}
Instead of setting up the workflow from scratch you can download the workflow here.

Load it into the browser page by clicking the Comfy Logo -> File -> Open -> Then choose the `unsloth_qwen_image_2512.json` file you just downloaded. It should look like the below:
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FqoxBnRlnYrmzLfZshE1Z%2FScreenshot%20from%202025-12-29%2014-37-00.png?alt=media&amp;token=1b1517b7-d44f-4e95-a5ed-759a4e0f74ec" alt="" width="254"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FIg48mzpraPTu1O16X1ay%2FScreenshot%20from%202025-12-30%2015-27-35.png?alt=media&amp;token=c761e48a-18cd-4291-9f29-3c09f621729d" alt="" width="563"><figcaption></figcaption></figure>

This workflow is based on the official ComfyUI published workflow except it uses the GGUF loader extension, and is simplified to illustrate text to image functionality.&#x20;

### #4. Inference

ComfyUI is highly customizable. You can mix models and create extremely complex pipelines. For a basic text to image setup we need to load the model, specify prompt and image details, and decide on a sampling strategy.&#x20;

#### **Upload Models + Set Prompt**

We already downloaded the models, so we just need to pick the correct ones. For Unet Loader pick `qwen-image-2512-Q4_K_M.gguf`, for CLIPLoader pick `Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf`, and for Load VAE pick `qwen_image_vae.safetensors`.&#x20;

{% hint style="info" %}
For more realistic results, skip keywords like “photorealistic” or “digital rendering” or “3d render” and use terms like “photograph” instead.
{% endhint %}

You can set any prompt you'd like, and also specify a negative prompt. The negative prompt helps by telling the model where to steer away from.

{% hint style="info" %}
For negative prompts, it’s best to use an NLP-style approach: describe in **natural language** what you *don’t* want in the image. Packing in too many keywords can hurt results instead of making it more specific.
{% endhint %}

#### **Image Size + Sampler Parameters**

The Qwen Image model series supports different image sizes. You can make rectangular shapes by setting the values of width and height. For sampler parameters, you can experiment with different samplers other than euler, and more or less sampling steps. The workflow has steps set to 40, but for quick tests 20 might be good enough. Change the `control after generate` setting from randomize to fixed if you want to see how different settings change outputs.

#### **Run**

Click Run and an image will be generated in about 1 minute (30 seconds for 20 steps). That output image can be saved. The interesting part is that the metadata for the entire comfy workflow is saved in the image. You can share and anyone can see how it was created by loading it in the UI.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F3fvZ7Y7dyOwwWfQXP4U9%2Funsloth_woman.png?alt=media&amp;token=bf131b12-be85-45c9-83bb-6a087decb8bb" alt="" width="188"><figcaption></figcaption></figure>

{% hint style="info" %}
If you're encountering blurry/bad images, raise shift to 12-13! solves most issues with bad outputs.
{% endhint %}

#### **Multi Reference Generation**

A key feature of Qwen-Image-Edit-2511 is multi reference generation where you can supply multiple images to use to help control generation. This time load the `unsloth_qwen_image_edit_2511.json`. We will use most of the same models but switching `qwen-image-2512-Q4_K_M.gguf` to `qwen-image-edit-2511-Q4_K_M.gguf` for the unet. The other difference this time are extra nodes to select images to reference, which we've downloaded earlier. You'll notice the prompt refers to both `image 1` and `image 2` which are prompt anchors for the images. Once loaded click Run, and you'll see an output that creates our two unique sloth characters together while preserving their likeness.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FmHqctMHBGGtyTMh7VcxN%2Funsloth_diffusion1.png?alt=media&amp;token=c66059ae-4015-4fea-9181-75566bc7f03d" alt="" width="188"><figcaption><p>Final result made from images to the right:</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FvRbCNdlnmXOyM84YaF0b%2Fsloth%20gpu%20square.png?alt=media&amp;token=7b5c14ae-b5d7-4554-86e5-f8e2480bbb39" alt="" width="188"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FwAYdYspdgZGbxFePXf06%2Fsloth%20on%20gpu.jpg?alt=media&amp;token=1031686f-1f0d-423a-80ae-77127aaaa0bd" alt="" width="188"><figcaption></figcaption></figure></div>

## 🤗 D**iffusers Tutorial**

We have also uploaded a [Dynamic 4-bit BitsandBytes](https://huggingface.co/unsloth/Qwen-Image-2512-unsloth-bnb-4bit) quantized version which can be run in Hugging Face's `diffusers` library. Once again, it uses Unsloth Dynamic where important layers are upcasted to higher precision.

Run `Qwen-Image-2512-unsloth-bnb-4bit` with the code below:

```python
from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained(
    "unsloth/Qwen-Image-2512-unsloth-bnb-4bit",
    torch_dtype=torch.bfloat16,
).to('cuda')

# uncomment if you run out of memory
# pipe.enable_model_cpu_offload() 

output = pipe(
    prompt="a kawaii sloth playing the drums",
    negative_prompt="blurry, unfocused",
    num_inference_steps=20,
    true_cfg_scale=4.0,
)

# Save output
image = output.images[0]
image.save('sample.png')
```

## 🎨 **stable-diffusion.cpp Tutorial**

If you want to run the model in stable-diffusion.cpp, you can follow our [step-by-step guide here](/docs/models/tutorials/qwen-image-2512/stable-diffusion.cpp).


# Run Qwen-Image-2512 in stable-diffusion.cpp Tutorial

Tutorial for using Qwen-Image-2512 in stable-diffusion.cpp.

**Qwen-Image-2512** is Qwen's new text-to-image foundational model and you can now run it on your local device via stable-diffusion.cpp. See below for instructions:

## 📖 stable-diffusion.cpp Tutorial

[stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) is an open-source library for efficient and local inference of diffusion image models written in pure C/C++.

To run, you don't need a GPU, just a CPU with RAM will work. For best results, ensure your total usable memory (RAM + VRAM / unified) is larger than the GGUF size; e.g. 4-bit (Q4\_K\_M) `unsloth/Qwen-Image-Edit-2512-GGUF` is 13.1 GB, so you should have 13.2+ GB of combined memory.

The tutorial will focus on machines with CUDA available, but instructions to build with on Apple or CPU only are similar and available in the repo.

### #1. Setup environment

We will be building from source so we need to first be sure your build software is installed

```bash
sudo apt update
sudo apt install -y git cmake build-essential pkg-config
```

{% hint style="info" %}
[Releases Page](https://github.com/leejet/stable-diffusion.cpp/releases) may have pre built binaries available for your hardware if you don't want to go through the build process.
{% endhint %}

Make sure CUDA environment variables are set:

```bash
export CUDA_HOME=/usr/local/cuda
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"
```

You can confirm if set correctly by running:

```bash
nvcc --version  // if not found install nvidia-cuda-toolkit
ldconfig -p | grep -E 'libcudart\.so|libcublas\.so'
```

We can now clone the repo and build:

{% code expandable="true" %}

```bash
git clone --recursive https://github.com/leejet/stable-diffusion.cpp
cd stable-diffusion.cpp

mkdir -p build
cd build

cmake .. -DCMAKE_BUILD_TYPE=Release -DSD_CUDA=ON
cmake --build . -j"$(nproc)"
```

{% endcode %}

Confirm sd-cli was built:

```bash
ls bin/sd-cli
```

### #2. Download Models

Diffusion models typically need 3 components. A Variational AutoEncoder (VAE) that encodes image pixel space to latent space, a text encoder to translate text to input embeddings, and the actual diffusion transformer. Both the diffusion model and text encoder can be GGUF format while we typically use safetensors for the vae. Let's download the models we will use:

```bash
cd .. 
mkdir models
mkdir outputs

## Diffusion Models
curl -L -C - -o models/qwen-image-2512-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen-Image-2512-GGUF/resolve/main/qwen-image-2512-Q4_K_M.gguf
curl -L -C - -o models/qwen-image-edit-2511-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF/resolve/main/qwen-image-edit-2511-Q4_K_M.gguf
 
## Text Encoder + VAE   
curl -L -C - -o models/Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf \
  https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF/resolve/main/Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf
curl -L -C - -o models/qwen_image_vae.safetensors \
  https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/vae/qwen_image_vae.safetensors

```

We are using Q4 GGUF variants, but you can try smaller or larger quant types depending on how much VRAM/RAM you have.

{% hint style="warning" %}
The format of the vae and diffusion model might be different than the diffusers checkpoints. Only use checkpoints that are compatible with stable-diffusion.cpp and ComfyUI.
{% endhint %}

#### Workflow and Hyperparameters

You can view our detailed [Run GGUFs in ComfyUI](/docs/blog/comfyui#workflow-and-hyperparameters-1) Guide.

### #3. Inference

We can now run the binary that we built. This is an example of a basic text to image command:

```bash
./build/bin/sd-cli --diffusion-model models/qwen-image-2512-Q4_K_M.gguf \
    --vae models/qwen_image_vae.safetensors \
    --llm models/Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf \
    --cfg-scale 2.5 --sampling-method euler -v --steps 40 \
    -H 1024 -W 1024 --diffusion-fa --flow-shift 3 \
    -p 'Aerial drone photograph of a vast field of bright yellow wildflowers with the text "Unsloth + Diffusion" spelled out in deep purple lavender flowers, sharp contrast between yellow and purple, natural organic letter shapes formed by flower beds, golden hour lighting, rolling countryside landscape, high altitude perspective looking straight down, photorealistic, 8K resolution'  \
    --offload-to-cpu -o outputs/unsloth_diffusion.png
```

{% hint style="success" %}
No need for `--offload-to-cpu` if you have enough VRAM.
{% endhint %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFd4q6TxbtyQPcK7zAOIK%2Funsloth_diffusion.png?alt=media&amp;token=182e8d80-f27e-4d86-9f46-c21d4d29840a" alt="" width="188"><figcaption></figcaption></figure>


# Devstral 2 - How to Run Guide

Guide for local running Mistral Devstral 2 models: 123B-Instruct-2512 and Small-2-24B-Instruct-2512.

Devstral 2 are Mistral’s new coding and agentic LLMs for software engineering, available in [24B](#devstral-small-2-24b) and [123B](#devstral-2-123b) sizes. The 123B model achieves SOTA in SWE-bench, coding, tool-calling and agent use-cases. The 24B model fits in 25GB RAM/VRAM and 123B fits in 128GB.

{% hint style="success" %}
**13th December 2025 Update**

**We’ve resolved issues in Devstral’s chat template, and results should be significantly better. The 24B & 123B have been updated. Also install the latest llama.cpp as at 13th Dec 2025!**
{% endhint %}

Devstral 2 supports vision capabilities, a 256k context window and uses the same architecture as [Ministral 3](/docs/models/tutorials/ministral-3). You can now run and **fine-tune** both models locally with Unsloth.

All Devstral 2 uploads use our Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) methodology, delivering the best performance on [Aider Polyglot](/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot) and 5-shot MMLU benchmarks.

<a href="#devstral-small-2-24b" class="button primary">Devstral-Small-2-24B</a><a href="#devstral-2-123b" class="button primary">Devstral-2-123B</a>

#### **Devstral 2 - Unsloth Dynamic** GGUFs:

| Devstral-Small-2-24B-Instruct-2512                                                                                    | Devstral-2-123B-Instruct-2512                                                                               |
| --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| [Devstral-Small-2-**24B**-Instruct-2512-GGUF](https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF) | [Devstral-2-**123B**-Instruct-2512-GGUF](https://huggingface.co/unsloth/Devstral-2-123B-Instruct-2512-GGUF) |

## 🖥️ **Running Devstral 2**

See our step-by-step guides for running [Devstral 24B](#devstral-small-2-24b) and the large [Devstral 123B](#devstral-2-123b) models. Both models support vision support but currently **vision is not supported** in llama.cpp

### :gear: Usage Guide

Here are the recommended settings for inference:

* <mark style="background-color:blue;">**Temperature \~0.15**</mark>
* Min\_P of 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1)
* **Use `--jinja` to enable the system prompt.**
* Max context length = 262,144
* Recommended minimum context: 16,384
* Install the latest llama.cpp since a [December 13th 2025 pull request](https://github.com/ggml-org/llama.cpp/pull/17945) fixes issues.

### :tophat:Devstral-Small-2-24B

The full precision (Q8) Devstral-Small-2-24B GGUF will fit in 25GB RAM/VRAM. Text only for now.

#### ✨ Run Devstral-Small-2-24B-Instruct-2512 in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:`Q4_K_XL`) is the quantization type. You can also directly pull from Hugging Face:

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q4_K_XL \
    --jinja -ngl 99 --ctx-size 16384 \
    --temp 0.15
```

3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD_Q4_K_XL` or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF",
    local_dir = "unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*", "*mmproj-F16*"], # For Q4_K_XL
)
```

4. Run the model in conversation mode:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/mmproj-F16.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.15 \
    --jinja
```

{% endcode %}

#### :eyes:Devstral and vision

1. To play with Devstral's image capabilities, let's first download a image like this [FP8 Reinforcement Learning with Unsloth](https://unsloth.ai/cgi/image/fp8grpolarge_KharloZxEEaHAY2X97CEX.png?width=3840\&quality=80\&format=auto) below:\
   ![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVKuEX57zjziskMFnAuhY%2Fimage.png?alt=media\&token=83ef2463-888e-48f1-acf2-55b05a544b5f)
2. We get the image via `wget https://unsloth.ai/cgi/image/fp8grpolarge_KharloZxEEaHAY2X97CEX.png?width=3840%26quality=80%26format=auto -O unsloth_fp8.png` which will save the image as "unsloth\_fp8.png"
3. Then load the image in via `/image unsloth_fp8.png` after the model is loaded as seen below:\
   ![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F8BYfTecTXWUh1mqGwLYQ%2Fimage.png?alt=media\&token=d43ab46d-a075-402d-9a60-fba2c3d25869)
4. We then prompt it `Describe this image` and get the below:<br>

   <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FQygUnKEB0sPfko5LH9ap%2Fimage.png?alt=media&amp;token=54a459e4-8f29-4e64-8fab-10e17156ad55" alt=""><figcaption></figcaption></figure>

### :truck:Devstral-2-123B

The full precision (Q8) Devstral-Small-2-123B GGUF will fit in 128GB RAM/VRAM. Text only for now.

#### :sparkles: Run Devstral-2-123B-Instruct-2512 Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}

2. You can directly pull from HuggingFace via:

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Devstral-2-123B-Instruct-2512-GGUF:UD-Q2_K_XL \
    --jinja -ngl 99 --ctx-size 16384 \
    --temp 0.15
```

2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD_Q4_K_XL` or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Devstral-2-123B-Instruct-2512-GGUF",
    local_dir = "unsloth/Devstral-2-123B-Instruct-2512-GGUF",
    allow_patterns = ["*UD-Q2_K_XL*", "*mmproj-F16*"],
)
```

3. Run the model in conversation mode:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Devstral-2-123B-Instruct-2512-GGUF/Devstral-2-123B-Instruct-2512-UD-Q2_K_XL.gguf \
    --mmproj unsloth/Devstral-2-123B-Instruct-2512-GGUF/mmproj-F16.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.15 \
    --jinja
```

{% endcode %}

## 🦥 Fine-tuning Devstral 2 with Unsloth

Just like [Ministral 3](/docs/models/tutorials/ministral-3), Unsloth supports Devstral 2 fine-tuning. Training is 2x faster, use 70% less VRAM and supports 8x longer context lengths. Devstral 2 fits comfortably in a 24GB VRAM L4 GPU.

Unfortunately, Devstral 2 slightly exceeds the memory limits of a 16GB VRAM, so fine-tuning it for free on Google Colab isn't possible for now. However, you *can* fine-tune the model for free using our [Kaggle notebook](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_\(24B\)-Reasoning-Conversational.ipynb\&accelerator=nvidiaTeslaT4), which offers access to dual GPUs. Just change the notebook's Magistral model name to the `unsloth/Devstral-Small-2-24B-Instruct-2512` model.

{% hint style="success" %}
We made free Unsloth notebooks to fine-tune Ministral 3, and directly supports Devstral 2, since they share the same architecture! Change the name to use the desired model.
{% endhint %}

* Ministral-3B-Instruct [Vision notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_\(3B\)_Vision.ipynb) (vision) (Change model name to Devstral 2)
* Ministral-3B-Instruct [GRPO notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_\(3B\)_Reinforcement_Learning_Sudoku_Game.ipynb) (Change model name to Devstral 2)

{% columns %}
{% column %}
Devstral Vision finetuning notebook

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb>" %}
{% endcolumn %}

{% column %}
Devstral Sudoku GRPO RL notebook

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_(3B)_Reinforcement_Learning_Sudoku_Game.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

### :sunglasses:Llama-server serving & deployment

To deploy Devstral 2 for production, we use `llama-server` In a new terminal say via tmux, deploy the model via:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/mmproj-F16.gguf \
    --alias "unsloth/Devstral-Small-2-24B-Instruct-2512" \
    --n-gpu-layers 999 \
    --prio 3 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

{% endcode %}

When you run the above, you will get:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FJSTkWDCcHk5DI6otb72X%2Fimage.png?alt=media&amp;token=b685008e-e9ad-4dea-8f1d-af3fdade8e3b" alt=""><figcaption></figcaption></figure>

Then in a new terminal, after doing `pip install openai`, do:

{% code overflow="wrap" %}

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Devstral-Small-2-24B-Instruct-2512",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
```

{% endcode %}

Which will simply print 4.

### :toolbox:Tool Calling with Devstral 2 Tutorial

After following [#llama-server-serving-and-deployment](#llama-server-serving-and-deployment "mention") we then can load up some tools and see Devstral in action! Let's make some tools - copy paste and execute them in Python.

{% code expandable="true" %}

```python
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def subtract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([
        "A long time ago in a galaxy far far away...",
        "There were 2 friends who loved sloths and code...",
        "The world was ending because every sloth evolved to have superhuman intelligence...",
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "subtract_number": subtract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [
    {
        "type": "function",
        "function": {
            "name": "add_number",
            "description": "Add two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "multiply_number",
            "description": "Multiply two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "subtract_number",
            "description": "Subtract two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "write_a_story",
            "description": "Writes a random story.",
            "parameters": {
                "type": "object",
                "properties": {},
                "required": [],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "terminal",
            "description": "Perform operations from the terminal.",
            "parameters": {
                "type": "object",
                "properties": {
                    "command": {
                        "type": "string",
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",
                    },
                },
                "required": ["command"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "python",
            "description": "Call a Python interpreter with some Python code that will be ran.",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {
                        "type": "string",
                        "description": "The Python code to run",
                    },
                },
                "required": ["code"],
            },
        },
    },
]
```

{% endcode %}

We then ask a simple question from a random list of possible messages to test the model:

{% code overflow="wrap" %}

```python
import random
messages = [{
    "role": "user",
    "content": [random.choice([
        {"type": "text", "text": "Could you write me a story ?"},
        {"type": "text", "text": "What is today's date plus 3 days?"},
        {"type": "text", "text": "Get the current time in nanoseconds."},
        {"type": "text", "text": "Create a Fibonacci function in Python and find fib(20)."},
    ])],
}]
```

{% endcode %}

We then use the below functions (copy and paste and execute) which will parse the function calls automatically - Devstral 2 might make multiple in tandem!

```python
temperature = 0.15
from openai import OpenAI
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
model_name = next(iter(openai_client.models.list())).id
print(f"Using model = {model_name}")
has_tool_calls = True
original_messages_len = len(messages)
while has_tool_calls:
    print(f"Current messages = {messages}")
    response = openai_client.chat.completions.create(
        model = model_name,
        messages = messages,
        temperature = temperature,
        tools = tools if tools else None,
        tool_choice = "auto" if tools else None,
    )
    tool_calls = response.choices[0].message.tool_calls or []
    content = response.choices[0].message.content or ""
    tool_calls_dict = [tc.to_dict() for tc in tool_calls] if tool_calls else tool_calls
    messages.append({"role": "assistant", "tool_calls": tool_calls_dict, "content": content,})
    for tool_call in tool_calls:
        fx, args, _id = tool_call.function.name, tool_call.function.arguments, tool_call.id
        out = MAP_FN[fx](**json.loads(args))
        messages.append({"role": "tool", "tool_call_id": _id, "name": fx, "content": str(out),})
    else:
        has_tool_calls = False
print(json.dumps(messages[original_messages_len:], indent = 2))
```

And after 1 minute, we get:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F3XsKXoOUJ1PAyrLfjZxH%2Fimage.png?alt=media&amp;token=ec1e125d-1326-4804-942f-2f41998099b6" alt=""><figcaption></figcaption></figure>

Or in JSON form:

```json
[
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "JviLK0wUveWguuKQHgZdFdYI2adu85jy",
        "function": {
          "arguments": "{}",
          "name": "write_a_story"
        },
        "type": "function"
      }
    ],
    "content": null
  },
  {
    "role": "tool",
    "tool_call_id": "JviLK0wUveWguuKQHgZdFdYI2adu85jy",
    "name": "write_a_story",
    "content": "A long time ago in a galaxy far far away..."
  },
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "In a distant galaxy, where the stars burned with an otherworldly glow, there was a planet named Eldoria. Eldoria was a world of stark contrasts\u2014vibrant cities nestled between towering mountains and vast deserts that stretched endlessly under twin suns. The people of Eldoria were known far and wide for their mastery of energy, particularly their legendary laser sabers.\n\nThese sabers were not mere weapons; they were extensions of their wielders' souls, forged from rare crystals that pulsed with the essence of the cosmos. Each saber was unique, its color and power reflecting the personality and spirit of its owner. The most skilled warriors, known as the Saberborn, could wield their sabers with such precision and grace that they seemed to dance with the very fabric of reality.\n\nAmong the Saberborn, there was a warrior named Kael. Kael was an outcast, a former guardian of the sacred Saber Temples who had been exiled for defying the council's orders. The council had sought to hoard the power of the laser sabers, using them to control the people of Eldoria. Kael believed that the sabers should be wielded by those who sought to protect, not dominate.\n\nOne day, Kael received a distress signal from a small village on the outskirts of the desert. The village was under attack by a rogue faction of Saberborn, led by a ruthless warlord named Vexis. Vexis sought to claim the village's ancient relic\u2014a crystal said to amplify the power of any laser saber tenfold. If Vexis succeeded, his army would become unstoppable, and Eldoria would fall into darkness.\n\nKael knew he had to act. He strapped on his saber\u2014a deep blue blade that hummed with the energy of the cosmos\u2014and set off across the desert. The journey was treacherous, with sandstorms and hidden traps set by Vexis's scouts. But Kael pressed on, driven by the memory of the people he had once sworn to protect.\n\nWhen he reached the village, the battle was already in full swing. Vexis's warriors wielded their sabers with brutal efficiency, cutting down defenders with ease. Kael leapt into the fray, his blue saber a blur of light as he disarmed and defeated one enemy after another. The villagers, seeing their savior arrive, rallied behind him, their own sabers flashing as they fought to reclaim their home.\n\nKael faced Vexis in the center of the village square. The warlord's saber was a sickly green, pulsing with dark energy. \"You are too late, Kael,\" Vexis sneered. \"The relic is mine, and with it, I will rule Eldoria.\" Kael stood his ground, his saber raised. \"Over my dead body,\" he replied.\n\nThe two warriors clashed, their sabers locking in a shower of sparks. Kael felt the raw power of the relic coursing through Vexis's blade, but he refused to back down. He channeled his own energy, his saber glowing brighter as he pushed back against Vexis's assault. With a final, desperate strike, Kael disarmed Vexis, sending his saber clattering to the ground.\n\nVexis snarled in defeat, but Kael did not kill him. Instead, he offered him a choice: \"Join me in protecting Eldoria, or leave and never return.\" Vexis, humbled and seeing the truth in Kael's words, chose to stand with him.\n\nWith Vexis's faction now allies, Kael and the villagers reclaimed the relic, using its power to restore balance to Eldoria. The Saber Temples were reformed, and the laser sabers were once again wielded by those who sought to protect, not control.\n\nKael's legend grew, and he became a symbol of hope for the people of Eldoria. His story reminded them that even in the darkest times, the light of courage and justice could prevail. And so, the Saberborn lived on, their laser sabers a beacon of strength and unity in a galaxy full of shadows."
  }
]
```


# Ministral 3 - How to Run Guide

Guide for Mistral Ministral 3 models, to run or fine-tune locally on your device

Mistral releases Ministral 3, their new multimodal models in Base, Instruct, and Reasoning variants, available in **3B**, **8B**, and **14B** sizes. They offer best-in-class performance for their size, and are fine-tuned for instruction and chat use cases. The multimodal models support **256K context** windows, multiple languages, native function calling, and JSON output.

The full unquantized 14B Ministral-3-Instruct-2512 model fits in **24GB RAM**/VRAM. You can now run, fine-tune and RL on all Ministral 3 models with Unsloth:

<a href="#run-ministral-3-tutorials" class="button primary">Run Ministral 3 Tutorials</a><a href="/docs/models/tutorials/ministral-3#fine-tuning" class="button primary">Fine-tuning Ministral 3</a>

We've also uploaded Mistral Large 3 [GGUFs here](https://huggingface.co/unsloth/Mistral-Large-3-675B-Instruct-2512-GGUF). For all Ministral 3 uploads (BnB, FP8), [see here](https://huggingface.co/collections/unsloth/ministral-3).

| Ministral-3-Instruct GGUFs:                                                                                                                                                                                                | Ministral-3-Reasoning GGUFs:                                                                                                                                                                                                  |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [3B](https://huggingface.co/unsloth/Ministral-3-3B-Instruct-2512-GGUF) • [8B](https://huggingface.co/unsloth/Ministral-3-8B-Instruct-2512-GGUF) • [14B](https://huggingface.co/unsloth/Ministral-3-14B-Instruct-2512-GGUF) | [3B](https://huggingface.co/unsloth/Ministral-3-3B-Reasoning-2512-GGUF) • [8B](https://huggingface.co/unsloth/Ministral-3-8B-Reasoning-2512-GGUF) • [14B](https://huggingface.co/unsloth/Ministral-3-14B-Reasoning-2512-GGUF) |

### ⚙️ Usage Guide

To achieve optimal performance for **Instruct**, Mistral recommends using lower temperatures such as `temperature = 0.15` or `0.1`<br>

For **Reasoning**, Mistral recommends `temperature = 0.7` and `top_p = 0.95`.

| Instruct:                     | Reasoning:          |
| ----------------------------- | ------------------- |
| `Temperature = 0.15` or `0.1` | `Temperature = 0.7` |
| `Top_P = default`             | `Top_P = 0.95`      |

**Adequate Output Length**: Use an output length of `32,768` tokens for most queries for the reasoning variant, and `16,384` for the instruct variant. You can increase the max output size for the reasoning model if necessary.

The maximum context length Ministral 3 can reach is `262,144`

The chat template format is found when we use the below:

{% code overflow="wrap" %}

```python
tokenizer.apply_chat_template([
    {"role" : "user", "content" : "What is 1+1?"},
    {"role" : "assistant", "content" : "2"},
    {"role" : "user", "content" : "What is 2+2?"}
    ], add_generation_prompt = True
)
```

{% endcode %}

#### Ministral *Reasoning* chat template:

{% code overflow="wrap" lineNumbers="true" %}

```
<s>[SYSTEM_PROMPT]# HOW YOU SHOULD THINK AND ANSWER

First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.

Your thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response to the user.[/THINK]Here, provide a self-contained response.[/SYSTEM_PROMPT][INST]What is 1+1?[/INST]2</s>[INST]What is 2+2?[/INST]
```

{% endcode %}

#### Ministral *Instruct* chat template:

{% code overflow="wrap" lineNumbers="true" expandable="true" %}

```
<s>[SYSTEM_PROMPT]You are Ministral-3-3B-Instruct-2512, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.
You power an AI assistant called Le Chat.
Your knowledge base was last updated on 2023-10-01.
The current date is {today}.

When you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
If the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. "What are some good restaurants around me?" => "Where are you?" or "When is the next flight to Tokyo" => "Where do you travel from?").
You are always very attentive to dates, in particular you try to resolve dates (e.g. "yesterday" is {yesterday}) and when asked about information at specific dates, you discard information that is at another date.
You follow these instructions in all languages, and always respond to the user in the language they use or request.
Next sections describe the capabilities that you have.

# WEB BROWSING INSTRUCTIONS

You cannot perform any web search or access internet to open URLs, links etc. If it seems like the user is expecting you to do so, you clarify the situation and ask the user to copy paste the text directly in the chat.

# MULTI-MODAL INSTRUCTIONS

You have the ability to read images, but you cannot generate images. You also cannot transcribe audio files or videos.
You cannot read nor transcribe audio files or videos.

# TOOL CALLING INSTRUCTIONS

You may have access to tools that you can use to fetch information or perform actions. You must use these tools in the following situations:

1. When the request requires up-to-date information.
2. When the request requires specific data that you do not have in your knowledge base.
3. When the request involves actions that you cannot perform without tools.

Always prioritize using tools to provide the most accurate and helpful response. If tools are not available, inform the user that you cannot perform the requested action at the moment.[/SYSTEM_PROMPT][INST]What is 1+1?[/INST]2</s>[INST]What is 2+2?[/INST]
```

{% endcode %}

## 📖 Run Ministral 3 Tutorials

Below are guides for the [Reasoning](#reasoning-ministral-3-reasoning-2512) and [Instruct](#instruct-ministral-3-instruct-2512) variants of the model.

### Instruct: Ministral-3-Instruct-2512

To achieve optimal performance for **Instruct**, Mistral recommends using lower temperatures such as `temperature = 0.15` or `0.1`

#### :sparkles: Llama.cpp: Run Ministral-3-14B-Instruct Tutorial

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
You can directly pull from Hugging Face via:

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Ministral-3-14B-Instruct-2512-GGUF:Q4_K_XL \
    --jinja -ngl 99 --ctx-size 32784 \
    --temp 0.15
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD_Q4_K_XL` or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Ministral-3-14B-Instruct-2512-GGUF",
    local_dir = "Ministral-3-14B-Instruct-2512-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

{% endstep %}
{% endstepper %}

### Reasoning: Ministral-3-Reasoning-2512

To achieve optimal performance for **Reasoning**, Mistral recommends using `temperature = 0.7` and `top_p = 0.95`.

#### :sparkles: Llama.cpp: Run Ministral-3-14B-Reasoning Tutorial

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub](https://github.com/ggml-org/llama.cpp). You can also use the build instructions below. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
You can directly pull from Hugging Face via:

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Ministral-3-14B-Reasoning-2512-GGUF:Q4_K_XL \
    --jinja -ngl 99 --ctx-size 32784 \
    --temp 0.6 --top-p 0.95
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD_Q4_K_XL` or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Ministral-3-14B-Reasoning-2512-GGUF",
    local_dir = "Ministral-3-14B-Reasoning-2512-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

{% endstep %}
{% endstepper %}

## 🛠️ Fine-tuning Ministral 3 <a href="#fine-tuning" id="fine-tuning"></a>

Unsloth now supports fine-tuning of all Ministral 3 models, including vision support. To train, you must use the latest 🤗Hugging Face `transformers` v5 and `unsloth` which includes our our recent [ultra long context](/docs/blog/500k-context-length-fine-tuning) support. The large 14B Ministral 3 model should fit on a free Colab GPU.

We made free Unsloth notebooks to fine-tune Ministral 3. Change the name to use the desired model.

* Ministral-3B-Instruct [Vision notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_\(3B\)_Vision.ipynb) (vision)
* Ministral-3B-Instruct [GRPO notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_\(3B\)_Reinforcement_Learning_Sudoku_Game.ipynb)

{% columns %}
{% column %}
Ministral Vision finetuning notebook

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb>" %}
{% endcolumn %}

{% column %}
Ministral Sudoku GRPO RL notebook

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_(3B)_Reinforcement_Learning_Sudoku_Game.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

### :sparkles:Reinforcement Learning (GRPO)

Unsloth now supports RL and GRPO for the Mistral models as well. As usual, they benefit from all of Unsloth's enhancements and tomorrow, we are going to release a notebook soon specifically for autonomously solving the sudoku puzzle.

* Ministral-3B-Instruct [GRPO notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_\(3B\)_Reinforcement_Learning_Sudoku_Game.ipynb)

**To use the latest version of Unsloth and transformers v5, update via:**

{% code overflow="wrap" %}

```
pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth unsloth_zoo
```

{% endcode %}

The goal is to auto generate strategies to complete Sudoku!

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F2qDbhHfpuhNAHOtIernm%2Fimage.png?alt=media&amp;token=9a3d4bb2-3994-4ec8-aeb8-16bc2bcb77c4" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLZlHHeAjoVAeO6juQDiC%2Fimage.png?alt=media&amp;token=45abbb30-b705-4eec-81fc-fb99dd0c2621" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

For the reward plots for Ministral, we get the below. We see it works well!

{% columns %}
{% column %}
![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FqpfPNKkSF2O1T0flshEi%2Funknown.png?alt=media\&token=a2f14139-bcab-40bf-a054-f189de5d23df)

![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fe8TBzOVVn5iYhlJ6nh63%2Funknown.png?alt=media\&token=520699f9-ffd0-43a5-a0ef-263fa678b4bd)
{% endcolumn %}

{% column %}
![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FudSxKSBuSOIXONrtarmp%2Funknown.png?alt=media\&token=beefcbce-67df-4ce2-92b8-3e0adc240df6)

![](https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FgwwlcVjMt9nqyqVC6xqD%2Funknown.png?alt=media\&token=b5b390b6-c9e6-4926-9a70-d4aa365caa86)
{% endcolumn %}
{% endcolumns %}


# DeepSeek-OCR: How to Run & Fine-tune

Guide on how to run and fine-tune DeepSeek-OCR locally.

**DeepSeek-OCR** is a 3B-parameter vision model for OCR and document understanding. It uses *context optical compression* to convert 2D layouts into vision tokens, enabling efficient long-context processing.

Capable of handling tables, papers, and handwriting, DeepSeek-OCR achieves 97% precision while using 10× fewer vision tokens than text tokens - making it 10× more efficient than text-based LLMs.

You can fine-tune DeepSeek-OCR to enhance its vision or language performance. In our Unsloth [**free fine-tuning notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_\(3B\).ipynb), we demonstrated a [88.26% improvement](#fine-tuning-deepseek-ocr) for language understanding.

<a href="#running-deepseek-ocr" class="button primary">Running DeepSeek-OCR</a><a href="#fine-tuning-deepseek-ocr" class="button primary">Fine-tuning DeepSeek-OCR</a>

> **Our model upload that enables fine-tuning + more inference support:** [**DeepSeek-OCR**](https://huggingface.co/unsloth/DeepSeek-OCR)

## 🖥️ **Running DeepSeek-OCR**

To run the model in [vLLM](#vllm-run-deepseek-ocr-tutorial) or [Unsloth](#unsloth-run-deepseek-ocr-tutorial), here are the recommended settings:

### :gear: Recommended Settings

DeepSeek recommends these settings:

* <mark style="background-color:blue;">**Temperature = 0.0**</mark>
* `max_tokens = 8192`
* `ngram_size = 30`
* `window_size = 90`

### 📖 vLLM: Run DeepSeek-OCR Tutorial

1. Obtain the latest `vLLM` via:

```bash
uv venv
source .venv/bin/activate
# Until v0.11.1 release, you need to install vLLM from nightly build
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
```

2. Then run the following code:

{% code overflow="wrap" %}

```python
from vllm import LLM, SamplingParams
from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor
from PIL import Image

# Create model instance
llm = LLM(
    model="unsloth/DeepSeek-OCR",
    enable_prefix_caching=False,
    mm_processor_cache_gb=0,
    logits_processors=[NGramPerReqLogitsProcessor]
)

# Prepare batched input with your image file
image_1 = Image.open("path/to/your/image_1.png").convert("RGB")
image_2 = Image.open("path/to/your/image_2.png").convert("RGB")
prompt = "<image>\nFree OCR."

model_input = [
    {
        "prompt": prompt,
        "multi_modal_data": {"image": image_1}
    },
    {
        "prompt": prompt,
        "multi_modal_data": {"image": image_2}
    }
]

sampling_param = SamplingParams(
    temperature=0.0,
    max_tokens=8192,
    # ngram logit processor args
    extra_args=dict(
        ngram_size=30,
        window_size=90,
        whitelist_token_ids={128821, 128822},  # whitelist: <td>, </td>
    ),
    skip_special_tokens=False,
)
# Generate output
model_outputs = llm.generate(model_input, sampling_param)

# Print output
for output in model_outputs:
    print(output.outputs[0].text)
```

{% endcode %}

### 🦥 Unsloth: Run DeepSeek-OCR Tutorial

1. Obtain the latest `unsloth` via `pip install --upgrade unsloth` . If you already have Unsloth, update it via `pip install --upgrade --force-reinstall --no-deps --no-cache-dir unsloth unsloth_zoo`
2. Then use the code below to run DeepSeek-OCR:

{% code overflow="wrap" %}

```python
from unsloth import FastVisionModel
import torch
from transformers import AutoModel
import os
os.environ["UNSLOTH_WARN_UNINITIALIZED"] = '0'

from huggingface_hub import snapshot_download
snapshot_download("unsloth/DeepSeek-OCR", local_dir = "deepseek_ocr")
model, tokenizer = FastVisionModel.from_pretrained(
    "./deepseek_ocr",
    load_in_4bit = False, # Use 4bit to reduce memory use. False for 16bit LoRA.
    auto_model = AutoModel,
    trust_remote_code = True,
    unsloth_force_compile = True,
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
)

prompt = "<image>\nFree OCR. "
image_file = 'your_image.jpg'
output_path = 'your/output/dir'
res = model.infer(tokenizer, prompt=prompt, image_file=image_file, output_path = output_path, base_size = 1024, image_size = 640, crop_mode=True, save_results = True, test_compress = False)
```

{% endcode %}

## 🦥 **Fine-tuning DeepSeek-OCR**

Unsloth supports fine-tuning of DeepSeek-OCR. Since the default model isn't runnable on the latest `transformers` version, we added changes from the [Stranger Vision HF](https://huggingface.co/strangervisionhf) team, to then enable inference. As usual, Unsloth trains DeepSeek-OCR 1.4x faster with 40% less VRAM and 5x longer context lengths - no accuracy degradation.\
\
We created two free DeepSeek-OCR Colab notebooks (with and without eval):

* DeepSeek-OCR: [Fine-tuning only notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_\(3B\).ipynb)
* DeepSeek-OCR: [Fine-tuning + Evaluation notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_\(3B\)-Eval.ipynb) (A100)

Fine-tuning DeepSeek-OCR on a 200K sample Persian dataset resulted in substantial gains in Persian text detection and understanding. We evaluated the base model against our fine-tuned version on 200 Persian transcript samples, observing an **88.26% absolute improvement** in Character Error Rate (CER). After only 60 training steps (batch size = 8), the mean CER decreased from **149.07%** to a mean of **60.81%**. This means the fine-tuned model is **57%** more accurate at understanding Persian.

You can replace the Persian dataset with your own to improve DeepSeek-OCR for other use-cases.\
\
For replica-table eval results, use our eval notebook above. For detailed eval results, see below:

### Fine-tuned Evaluation Results:

{% columns fullWidth="true" %}
{% column %}
**DeepSeek-OCR Baseline**

Mean Baseline Model Performance: 149.07% CER for this eval set!

```
============================================================
Baseline Model Performance
============================================================
Number of samples: 200
Mean CER: 149.07%
Median CER: 80.00%
Std Dev: 310.39%
Min CER: 0.00%
Max CER: 3500.00%
============================================================

 Best Predictions (Lowest CER):

Sample 5024 (CER: 0.00%)
Reference:  چون هستی خیلی زیاد...
Prediction: چون هستی خیلی زیاد...

Sample 3517 (CER: 0.00%)
Reference:  تو ایران هیچوقت از اینها وجود نخواهد داشت...
Prediction: تو ایران هیچوقت از اینها وجود نخواهد داشت...

Sample 9949 (CER: 0.00%)
Reference:  کاش میدونستم هیچی بیخیال...
Prediction: کاش میدونستم هیچی بیخیال...

 Worst Predictions (Highest CER):

Sample 11155 (CER: 3500.00%)
Reference:  خسو...
Prediction: \[ \text{CH}_3\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}...

Sample 13366 (CER: 1900.00%)
Reference:  مشو...
Prediction: \[\begin{align*}\underline{\mathfrak{su}}_0\end{align*}\]...

Sample 10552 (CER: 1014.29%)
Reference:  هیییییچ...
Prediction: e
```

{% endcolumn %}

{% column %}
**DeepSeek-OCR Fine-tuned**

With 60 steps, we reduced CER from 149.07% to 60.43% (89% CER improvement)

<pre><code><strong>============================================================
</strong>Fine-tuned Model Performance
============================================================
Number of samples: 200
Mean CER: 60.43%
Median CER: 50.00%
Std Dev: 80.63%
Min CER: 0.00%
Max CER: 916.67%
============================================================

 Best Predictions (Lowest CER):

Sample 301 (CER: 0.00%)
Reference:  باشه بابا تو لاکچری، تو خاص، تو خفن...
Prediction: باشه بابا تو لاکچری، تو خاص، تو خفن...

Sample 2512 (CER: 0.00%)
Reference:  از شخص حاج عبدالله زنجبیلی میگیرنش...
Prediction: از شخص حاج عبدالله زنجبیلی میگیرنش...

Sample 2713 (CER: 0.00%)
Reference:  نمی دونم والا تحمل نقد ندارن ظاهرا...
Prediction: نمی دونم والا تحمل نقد ندارن ظاهرا...

 Worst Predictions (Highest CER):

Sample 14270 (CER: 916.67%)
Reference:  ۴۳۵۹۴۷۴۷۳۸۹۰...
Prediction: پروپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپیپریپریپریپریپریپریپریپریپریپریپریپریپریپر...

Sample 3919 (CER: 380.00%)
Reference:  ۷۵۵۰۷۱۰۶۵۹...
Prediction: وادووووووووووووووووووووووووووووووووووو...

Sample 3718 (CER: 333.33%)
Reference:  ۳۲۶۷۲۲۶۵۵۸۴۶...
Prediction: پُپُسوپُسوپُسوپُسوپُسوپُسوپُسوپُسوپُسوپُ...
</code></pre>

{% endcolumn %}
{% endcolumns %}

An example from the 200K Persian dataset we used (you may use your own), showing the image on the left and the corresponding text on the right.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-2afa75f90055db094d5cae1c635b200c05e97aac%2FScreenshot%202025-11-04%20at%206.10.16%E2%80%AFAM.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>


# Kimi K2 Thinking: Run Locally Guide

Guide on running Kimi-K2-Thinking and Kimi-K2 on your own local device!

{% hint style="success" %}
Kimi-K2-Thinking got released. Read our [Thinking guide](#kimi-k2-thinking-guide) or access [GGUFs here](https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF).

We also collaborated with the Kimi team on [**system prompt fix**](#tokenizer-quirks-and-bug-fixes) for Kimi-K2-Thinking.
{% endhint %}

Kimi-K2 and **Kimi-K2-Thinking** achieve SOTA performance in knowledge, reasoning, coding, and agentic tasks. The full 1T parameter models from Moonshot AI requires 1.09TB of disk space, while the quantized **Unsloth Dynamic 1.8-bit** version reduces this to just 230GB (-80% size)**:** [**Kimi-K2-GGUF**](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF)

You can also now run our [**Kimi-K2-Thinking** GGUFs](https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF).

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA [Aider Polyglot](/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot) and 5-shot MMLU performance. See how our Dynamic 1–2 bit GGUFs perform on [coding benchmarks here](/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot).

<a href="#kimi-k2-thinking-guide" class="button primary">Run Thinking</a><a href="#kimi-k2-instruct-guide" class="button primary">Run Instruct</a>

### :gear: Recommended Requirements

{% hint style="info" %}
You need **247GB of disk space** to run the 1bit quant!

The only requirement is **`disk space + RAM + VRAM ≥ 247GB`**. That means you do not need to have that much RAM or VRAM (GPU) to run the model, but it will be much slower.
{% endhint %}

The 1.8-bit (UD-TQ1\_0) quant will fit in a 1x 24GB GPU (with all MoE layers offloaded to system RAM or a fast disk). Expect around \~1-2 tokens/s with this setup if you have bonus 256GB RAM as well. The full Kimi K2 Q8 quant is 1.09TB in size and will need at least 8 x H200 GPUs.

For optimal performance you will need at least **247GB unified memory or 247GB combined RAM+VRAM** for 5+ tokens/s. If you have less than 247GB combined RAM+VRAM, then the speed of the model will definitely take a hit.

**If you do not have 247GB of RAM+VRAM, no worries!** llama.cpp inherently has **disk offloading**, so through mmaping, it'll still work, just be slower - for example before you might get 5 to 10 tokens / second, now it's under 1 token.

We suggest using our **UD-Q2\_K\_XL (360GB)** quant to balance size and accuracy!

{% hint style="success" %}
For the best performance, have your VRAM + RAM combined = the size of the quant you're downloading. If not, it'll still work via disk offloading, just it'll be slower!
{% endhint %}

## 💭Kimi-K2-Thinking Guide

Kimi-K2-Thinking should generally follow the same instructions as the Instruct model, with a few key differences, particularly in areas such as settings and the chat template.

{% hint style="success" %}
**To run the model in full precision, you only need to use the 4-bit or 5-bit Dynamic GGUFs (e.g. UD\_Q4\_K\_XL) because the model was originally released in INT4 format.**

You can choose a higher-bit quantization just to be safe in case of small quantization differences, but in most cases this is unnecessary.
{% endhint %}

### 🌙 Official Recommended Settings:

According to [Moonshot AI](https://huggingface.co/moonshotai/Kimi-K2-Thinking), these are the recommended settings for Kimi-K2-Thinking inference:

* Set the <mark style="background-color:green;">**temperature 1.0**</mark> to reduce repetition and incoherence.
* Suggested context length = 98,304 (up to 256K)
* Note: Using different tools may require different settings

{% hint style="info" %}
We recommend setting <mark style="background-color:green;">**min\_p to 0.01**</mark> to suppress the occurrence of unlikely tokens with low probabilities.
{% endhint %}

For example given a user message of "What is 1+1?", we get:

{% code overflow="wrap" %}

```
<|im_system|>system<|im_middle|>You are Kimi, an AI assistant created by Moonshot AI.<|im_end|><|im_user|>user<|im_middle|>What is 1+1?<|im_end|><|im_assistant|>assistant<|im_middle|>
```

{% endcode %}

### ✨ Run Kimi K2 Thinking in llama.cpp

{% hint style="success" %}
You can now use the latest update of [llama.cpp](https://github.com/ggml-org/llama.cpp) to run the model:
{% endhint %}

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:UD-TQ1\_0) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location.

```bash
export LLAMA_CACHE="unsloth/Kimi-K2-Thinking-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Kimi-K2-Thinking-GGUF:UD-TQ1_0 \
    --n-gpu-layers 99 \
    --temp 1.0 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

3. The above will use around 8GB of GPU memory. If you have around 360GB of combined GPU memory, remove `-ot ".ffn_.*_exps.=CPU"` to get maximum speed!

{% hint style="info" %}
Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). We recommend using our 2bit dynamic quant UD-Q2\_K\_XL to balance size and accuracy. All versions at: [huggingface.co/unsloth/Kimi-K2-Thinking-GGUF](https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF)

{% code overflow="wrap" %}

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Kimi-K2-Thinking-GGUF",
    local_dir = "unsloth/Kimi-K2-Thinking-GGUF",
    allow_patterns = ["*UD-TQ1_0*"], # Use "*UD-Q2_K_XL*" for Dynamic 2bit (381GB)
)
```

{% endcode %}

{% hint style="info" %}
If you find that downloads get stuck at 90 to 95% or so, please see <https://docs.unsloth.ai/basics/troubleshooting-and-faqs#downloading-gets-stuck-at-90-to-95>
{% endhint %}

4. Run any prompt.
5. Edit `--threads -1` for the number of CPU threads (be default it's set to the maximum CPU threads), `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Set it to 99 combined with MoE CPU offloading to get the best performance. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Kimi-K2-Thinking-GGUF/UD-TQ1_0/Kimi-K2-Thinking-UD-TQ1_0-00001-of-00006.gguf \
    --n-gpu-layers 99 \
    --temp 1.0 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endcode %}

### :thinking:**No Thinking Tags?**

You may notice that there are no *thinking* tags when you run the model. This is normal and intended behavior.

In your `llama.cpp` script, make sure to include the `--special` flag at the very end of your command. Once you do, you’ll see the `<think>` token appear as expected.

You might also see every answer end with `<|im_end|>`. This is normal as `<|im_end|>` is a special token that appears when printing special tokens. If you’d like to hide it, you can set `<|im_end|>` as a stop string in your settings.

### ✨ Deploy with llama-server and OpenAI's completion library

After installing llama.cpp as per [#run-kimi-k2-thinking-in-llama.cpp](#run-kimi-k2-thinking-in-llama.cpp "mention"), you can use the below to launch an OpenAI compatible server:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/Kimi-K2-Thinking-GGUF/UD-TQ1_0/Kimi-K2-Thinking-UD-TQ1_0-00001-of-00006.gguf \
    --alias "unsloth/Kimi-K2-Thinking" \
    -fa on \
    --n-gpu-layers 999 \
    -ot ".ffn_.*_exps.=CPU" \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

{% endcode %}

Then use OpenAI's Python library after `pip install openai` :

```python
from openai import OpenAI
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Kimi-K2-Thinking",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
```

## :mag:Tokenizer quirks and bug fixes

**7th November 2025: We notified the Kimi team, and fixed the default system prompt of** `You are Kimi, an AI assistant created by Moonshot AI.` **not appearing on the first user prompt!** See <https://huggingface.co/moonshotai/Kimi-K2-Thinking/discussions/12>

Huge thanks to the Moonshot Kimi team for their extremely fast response time to our queries and fixing the issue ASAP!

**16th July 2025: Kimi K2 updated their tokenizer to enable multiple tool calls** as per <https://x.com/Kimi_Moonshot/status/1945050874067476962>

**18th July 2025: We fixed a system prompt - Kimi tweeted about our fix as well here:** <https://x.com/Kimi_Moonshot/status/1946130043446690030>. The fix was described here as well: <https://huggingface.co/moonshotai/Kimi-K2-Instruct/discussions/28>

If you have the old checkpoints downloaded - now worries - simply download the first GGUF split which was changed. OR if you do not want to download any new files do:

```bash
wget https://huggingface.co/unsloth/Kimi-K2-Instruct/raw/main/chat_template.jinja
./llama.cpp ... --chat-template-file /dir/to/chat_template.jinja
```

The Kimi K2 tokenizer was interesting to play around with - <mark style="background-color:green;">**it's mostly similar in action to GPT-4o's tokenizer**</mark>! We first see in the [tokenization\_kimi.py](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/tokenization_kimi.py) file the following regular expression (regex) that Kimi K2 uses:

```python
pat_str = "|".join(
    [
        r"""[\p{Han}]+""",
        r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
        r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
        r"""\p{N}{1,3}""",
        r""" ?[^\s\p{L}\p{N}]+[\r\n]*""",
        r"""\s*[\r\n]+""",
        r"""\s+(?!\S)""",
        r"""\s+""",
    ]
)
```

After careful inspection, we find Kimi K2 is nearly identical to GPT-4o's tokenizer regex which can be found in [llama.cpp's source code](https://github.com/ggml-org/llama.cpp/blob/55c509daf51d25bfaee9c8b8ce6abff103d4473b/src/llama-vocab.cpp#L400).

{% code overflow="wrap" %}

```
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+
```

{% endcode %}

Both tokenize numbers into groups of 1 to 3 numbers (9, 99, 999), and use similar patterns. The only difference looks to be the handling of "Han" or Chinese characters, which Kimi's tokenizer deals with more. [The PR](https://github.com/ggml-org/llama.cpp/pull/14654) by <https://github.com/gabriellarson> handles these differences well after some [discussions here](https://github.com/ggml-org/llama.cpp/issues/14642#issuecomment-3067324745).

<mark style="background-color:green;">**We also find the correct EOS token should not be \[EOS], but rather <|im\_end|>, which we have also fixed in our model conversions.**</mark>

## 🌝Kimi-K2-Instruct Guide

Step-by-step guide on running the Instruct Kimi K2 models including Kimi K2 0905 - the September 5 update.

### 🌙 Official Recommended Settings:

According to [Moonshot AI](https://huggingface.co/moonshotai/Kimi-K2-Instruct), these are the recommended settings for Kimi K2 inference:

* Set the <mark style="background-color:green;">**temperature 0.6**</mark> to reduce repetition and incoherence.
* Original default system prompt is:

  ```
  You are a helpful assistant
  ```
* (Optional) Moonshot also suggests the below for the system prompt:

  ```
  You are Kimi, an AI assistant created by Moonshot AI.
  ```

{% hint style="success" %}
We recommend setting <mark style="background-color:green;">**min\_p to 0.01**</mark> to suppress the occurrence of unlikely tokens with low probabilities.
{% endhint %}

### :1234: Chat template and prompt format

Kimi Chat does use a BOS (beginning of sentence token). The system, user and assistant roles are all enclosed with `<|im_middle|>` which is interesting, and each get their own respective token `<|im_system|>, <|im_user|>, <|im_assistant|>`.

{% code overflow="wrap" %}

```python
<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|><|im_user|>user<|im_middle|>What is 1+1?<|im_end|><|im_assistant|>assistant<|im_middle|>2<|im_end|>
```

{% endcode %}

To separate the conversational boundaries (you must remove each new line), we get:

{% code overflow="wrap" %}

```
<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|>
<|im_user|>user<|im_middle|>What is 1+1?<|im_end|>
<|im_assistant|>assistant<|im_middle|>2<|im_end|>
```

{% endcode %}

### :floppy\_disk: Model uploads

**ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and reasoning tasks.

<table data-full-width="false"><thead><tr><th>MoE Bits</th><th>Type + Link</th><th>Disk Size</th><th>Details</th></tr></thead><tbody><tr><td>1.66bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-TQ1_0">UD-TQ1_0</a></td><td><strong>245GB</strong></td><td>1.92/1.56bit</td></tr><tr><td>1.78bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ1_S">UD-IQ1_S</a></td><td><strong>281GB</strong></td><td>2.06/1.56bit</td></tr><tr><td>1.93bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ1_M">UD-IQ1_M</a></td><td><strong>304GB</strong></td><td>2.5/2.06/1.56</td></tr><tr><td>2.42bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ2_XXS">UD-IQ2_XXS</a></td><td><strong>343GB</strong></td><td>2.5/2.06bit</td></tr><tr><td>2.71bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q2_K_XL">UD-Q2_K_XL</a></td><td><strong>381GB</strong></td><td>3.5/2.5bit</td></tr><tr><td>3.12bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-IQ3_XXS">UD-IQ3_XXS</a></td><td><strong>417GB</strong></td><td>3.5/2.06bit</td></tr><tr><td>3.5bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q3_K_XL">UD-Q3_K_XL</a></td><td><strong>452GB</strong></td><td>4.5/3.5bit</td></tr><tr><td>4.5bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q4_K_XL">UD-Q4_K_XL</a></td><td><strong>588GB</strong></td><td>5.5/4.5bit</td></tr><tr><td>5.5bit</td><td><a href="https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF/tree/main/UD-Q5_K_XL">UD-Q5_K_XL</a></td><td><strong>732GB</strong></td><td>6.5/5.5bit</td></tr></tbody></table>

We've also uploaded versions in [BF16 format](https://huggingface.co/unsloth/Kimi-K2-Instruct-BF16).

### ✨ Run Instruct in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:UD-IQ1\_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location.\ <mark style="background-color:green;">**To run the new September 2025 update for the model, change the model name from 'Kimi-K2-Instruct' to 'Kimi-K2-Instruct-0905'.**</mark>

{% hint style="info" %}
Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

```bash
export LLAMA_CACHE="unsloth/Kimi-K2-Instruct-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Kimi-K2-Instruct-GGUF:TQ1_0 \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-TQ1_0`(dynamic 1.8bit quant) or other quantized versions like `Q2_K_XL` . We <mark style="background-color:green;">**recommend using our 2bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>. More versions at: [huggingface.co/unsloth/Kimi-K2-Instruct-GGUF](https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF)

{% code overflow="wrap" %}

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Kimi-K2-Instruct-GGUF",
    local_dir = "unsloth/Kimi-K2-Instruct-GGUF",
    allow_patterns = ["*UD-TQ1_0*"], # Dynamic 1bit (281GB) Use "*UD-Q2_K_XL*" for Dynamic 2bit (381GB)
)
```

{% endcode %}

{% hint style="info" %}
If you find that downloads get stuck at 90 to 95% or so, please see <https://docs.unsloth.ai/basics/troubleshooting-and-faqs#downloading-gets-stuck-at-90-to-95>
{% endhint %}

4. Run any prompt.
5. Edit `--threads -1` for the number of CPU threads (be default it's set to the maximum CPU threads), `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Set it to 99 combined with MoE CPU offloading to get the best performance. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Kimi-K2-Instruct-GGUF/UD-TQ1_0/Kimi-K2-Instruct-UD-TQ1_0-00001-of-00005.gguf \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endcode %}

### :bird: Flappy Bird + other tests <a href="#heptagon-test" id="heptagon-test"></a>

We introduced the Flappy Bird test when our 1.58bit quants for DeepSeek R1 were provided. We found Kimi K2 one of the only models to one-shot all our tasks including this one, [Heptagon ](/docs/models/tutorials/deepseek-r1-0528-how-to-run-locally#heptagon-test)and others tests even at 2-bit. The goal is to ask the LLM to create a Flappy Bird game but following some specific instructions:

{% code overflow="wrap" %}

```
Create a Flappy Bird game in Python. You must include these things:
1. You must use pygame.
2. The background color should be randomly chosen and is a light shade. Start with a light blue color.
3. Pressing SPACE multiple times will accelerate the bird.
4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.
5. Place on the bottom some land colored as dark brown or yellow chosen randomly.
6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.
7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.
8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.
The final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.
```

{% endcode %}

You can also test the dynamic quants via the Heptagon Test as per [r/Localllama](https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/) which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape.

<figure><img src="https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F2O72oTw5yPUbcxXjDNKS%252Fsnapshot.jpg%3Falt%3Dmedia%26token%3Dce852f9f-20ee-4b93-9d7b-1a5f211b9e04&#x26;width=768&#x26;dpr=4&#x26;quality=100&#x26;sign=55d1134d&#x26;sv=2" alt="" width="563"><figcaption></figcaption></figure>

The goal is to make the heptagon spin, and the balls in the heptagon should move. The prompt is below:

{% code overflow="wrap" %}

```
Write a Python program that shows 20 balls bouncing inside a spinning heptagon:\n- All balls have the same radius.\n- All balls have a number on it from 1 to 20.\n- All balls drop from the heptagon center when starting.\n- Colors are: #f8b862, #f6ad49, #f39800, #f08300, #ec6d51, #ee7948, #ed6d3d, #ec6800, #ec6800, #ee7800, #eb6238, #ea5506, #ea5506, #eb6101, #e49e61, #e45e32, #e17b34, #dd7a56, #db8449, #d66a35\n- The balls should be affected by gravity and friction, and they must bounce off the rotating walls realistically. There should also be collisions between balls.\n- The material of all the balls determines that their impact bounce height will not exceed the radius of the heptagon, but higher than ball radius.\n- All balls rotate with friction, the numbers on the ball can be used to indicate the spin of the ball.\n- The heptagon is spinning around its center, and the speed of spinning is 360 degrees per 5 seconds.\n- The heptagon size should be large enough to contain all the balls.\n- Do not use the pygame library; implement collision detection algorithms and collision response etc. by yourself. The following Python libraries are allowed: tkinter, math, numpy, dataclasses, typing, sys.\n- All codes should be put in a single Python file.
```

{% endcode %}


# GLM-4.6: Run Locally Guide

A guide on how to run Z.ai GLM-4.6 and GLM-4.6V-Flash model on your own local device!

GLM-4.6 and **GLM-4.6V-Flash** are the latest reasoning models from **Z.ai**, achieving SOTA performance on coding and agent benchmarks while offering improved conversational chats. [**GLM-4.6V-Flash**](#glm-4.6v-flash) **the smaller 9B model was released in December, 2025 and you can run it now too.**

The full 355B parameter model requires **400GB** of disk space, while the Unsloth Dynamic 2-bit GGUF reduces the size to **135GB** (-**75%)**. [**GLM-4.6-GGUF**](https://huggingface.co/unsloth/GLM-4.6-GGUF)

{% hint style="success" %}
For best performance, make sure your total available memory (VRAM + system RAM) exceeds the size of the quantized model file you’re downloading. If it doesn’t, llama.cpp can still run via SSD/HDD offloading, but inference will be slower.
{% endhint %}

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and Aider performance, meaning you can run & fine-tune quantized GLM LLMs with minimal accuracy loss.

**Tutorials navigation:**

<a href="#glm-4.6v-flash" class="button secondary">Run GLM-4.6V-Flash</a><a href="#glm-4.6" class="button secondary">Run GLM-4.6</a>

### :bug:Unsloth Chat Template & bug fixes

One of the significant fixes we did addresses an issue with prompting GGUFs, where the second prompt wouldn’t work. We fixed this issue however, this problem still persists in GGUFs without our fixes. For example, when using any non-Unsloth GLM-4.6 GGUF, the first conversation works fine, but the second one breaks.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-f1a25a7b2dbbabd5d04d079ae4dcf352bc326964%2Ftool-calling-on-glm-4-6-with-unsloths-ggufs-v0-oys0k2088nuf1.webp?alt=media" alt="" width="563"><figcaption></figcaption></figure>

We’ve resolved this in our chat template, so when using our version, conversations beyond the second (third, fourth, etc.) work without any errors. There are still some issues with tool-calling, which we haven’t fully investigated yet due to bandwidth limitations. We’ve already informed the GLM team about these remaining issues.

### :mag\_right:GLM 4.6V Flash quirks and fixes

{% hint style="info" %}
GLM-4.6V-Flash may reason and output in Chinese. This is not unique to our quants, but a quirk to the model. Use a system prompt of "Respond in English and reason in English" to force reasoning and outputs in English!
{% endhint %}

We tested other quant provider's BF16 and Q8\_0 quants, and all seem to reason in Chinese. For example 2 separate quants on seed 3407 and on the same prompt "Create a Flappy Bird game in Python" show reasoning in Chinese:

{% columns %}
{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FiQxOQlwJSuun6suWO5KR%2Fimage.png?alt=media&amp;token=f9228737-46ae-4077-825e-cff965003b20" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FhhQ4dOjfqBHtSOkL4Ldg%2Fimage.png?alt=media&amp;token=a6cd50fe-179a-44c2-a31d-987e78901985" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

By using a system prompt of "Reason in English" via `--system-prompt "Respond in English"` in llama.cpp, ie like below:

```bash
./llama.cpp/llama-cli -hf unsloth/GLM-4.6V-Flash-GGUF:BF16 \
    --jinja --temp 0.8 --top-p 0.6 --top-k 2 --repeat-penalty 1.1 --min-p 0.0 --seed 3407 \
    --prompt "Create a Flappy Bird game in Python" --system-prompt "Respond in English"
```

We get reasoning in Chinese, but outputs in English. We also ask a follow up of "What is 1+1" and get English only:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FIhIYFR5tIDa3Dc3FyURO%2Fimage.png?alt=media&amp;token=d79b38d0-a1c8-4c29-bd5a-116a39d9e85a" alt="" width="375"><figcaption></figcaption></figure>

And finally, by using a system prompt of "Respond in English and reason in English" via `--system-prompt "Respond in English and reason in English"` in llama.cpp, ie like below:

```bash
./llama.cpp/llama-cli -hf unsloth/GLM-4.6V-Flash-GGUF:BF16 \
    --jinja --temp 0.8 --top-p 0.6 --top-k 2 --repeat-penalty 1.1 --min-p 0.0 --seed 3407 \
    --prompt "Create a Flappy Bird game in Python" \
    --system-prompt "Respond in English and reason in English"
```

We get reasoning in English and outputs in English! We also ask a follow up of "What is 1+1" and get English only:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F55ykzBc3su8TzIePlh3t%2Fimage.png?alt=media&amp;token=07b9f59a-7879-4831-86a0-4ed234c4ba99" alt=""><figcaption></figcaption></figure>

## :gear: Usage Guide

The 2-bit dynamic quant UD-Q2\_K\_XL uses 135GB of disk space - this works well in a **1x24GB card and 128GB of RAM** with MoE offloading. The 1-bit UD-TQ1 GGUF also **works natively in Ollama**!

{% hint style="info" %}
You must use `--jinja` for llama.cpp quants - this uses our [fixed chat templates](#chat-template-bug-fixes) and enables the correct template! You might get incorrect results if you do not use `--jinja`
{% endhint %}

The 4-bit quants will fit in a 1x 40GB GPU (with MoE layers offloaded to RAM). Expect around 5 tokens/s with this setup if you have bonus 165GB RAM as well. It is recommended to have at least 205GB RAM to run this 4-bit. For optimal performance you will need at least 205GB unified memory or 205GB combined RAM+VRAM for 5+ tokens/s. To learn how to increase generation speed and fit longer contexts, [read here](#improving-generation-speed).

{% hint style="success" %}
Though not a must, for best performance, have your VRAM + RAM combined equal to the size of the quant you're downloading. If not, hard drive / SSD offloading will work with llama.cpp, just inference will be slower.
{% endhint %}

### Recommended Settings

According to Z.ai, there are different settings for GLM-4.6V-Flash & GLM-4.6 inference:

| GLM-4.6V-Flash                                                              | GLM-4.6                                                                                 |
| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| <mark style="background-color:green;">**temperature = 0.8**</mark>          | <mark style="background-color:green;">**temperature = 1.0**</mark>                      |
| <mark style="background-color:green;">**top\_p = 0.6**</mark> (recommended) | <mark style="background-color:green;">**top\_p = 0.95**</mark> (recommended for coding) |
| <mark style="background-color:green;">**top\_k = 2**</mark> (recommended)   | <mark style="background-color:green;">**top\_k = 40**</mark> (recommended for coding)   |
| **128K context length** or less                                             | **200K context length** or less                                                         |
| **repeat\_penalty = 1.1**                                                   |                                                                                         |
| **max\_generate\_tokens = 16,384**                                          | **max\_generate\_tokens = 16,384**                                                      |

* Use `--jinja` for llama.cpp variants - we **fixed some chat template issues as well!**

## Run GLM-4.6 Tutorials:

See our step-by-step guides for running [GLM-4.6V-Flash](#glm-4.6v-flash) and the large [GLM-4.6](#glm-4.6) models.

### GLM-4.6V-Flash

{% hint style="success" %}
**NEW as of Dec 16, 2025: GLM-4.6-V is now updated with vision support!**
{% endhint %}

#### ✨ Run in llama.cpp

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub](https://github.com/ggml-org/llama.cpp). You can also use the build instructions below. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:Q8\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 128K context length.

```bash
export LLAMA_CACHE="unsloth/GLM-4.6V-Flash-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/GLM-4.6V-Flash-GGUF:UD-Q8_K_XL \
    --n-gpu-layers 99 \
    --jinja \
    --ctx-size 16384 \
    --flash-attn on \
    --temp 0.8 \
    --top-p 0.6 \
    --top-k 2 \
    --repeat-penalty 1.1 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-`Q4\_K\_XL (dynamic 4bit quant) or other quantized versions like `Q8_K_XL` .

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/GLM-4.6V-Flash-GGUF",
    local_dir = "unsloth/GLM-4.6V-Flash-GGUF",
    allow_patterns = ["*UD-Q8_K_XL*"],
)
```

{% endstep %}
{% endstepper %}

### GLM-4.6

#### :llama: Run in Ollama

{% stepper %}
{% step %}
Install `ollama` if you haven't already! To run more variants of the model, [see here](https://unsloth.ai/docs/models/tutorials/pages/eK0BfjMHNrfvfe4HI6Pk#run-in-llama.cpp).

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

{% endstep %}

{% step %}
Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```
OLLAMA_MODELS=unsloth ollama serve &

OLLAMA_MODELS=unsloth ollama run hf.co/unsloth/GLM-4.6-GGUF:TQ1_0
```

{% endstep %}

{% step %}
To run other quants, you need to first merge the GGUF split files into 1 like the code below. Then you will need to run the model locally.

```bash
./llama.cpp/llama-gguf-split --merge \
  GLM-4.6-GGUF/GLM-4.6-UD-Q2_K_XL/GLM-4.6-UD-Q2_K_XL-00001-of-00003.gguf \
	merged_file.gguf
```

```bash
OLLAMA_MODELS=unsloth ollama serve &

OLLAMA_MODELS=unsloth ollama run merged_file.gguf
```

{% endstep %}
{% endstepper %}

#### ✨ Run in llama.cpp

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:Q2\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 128K context length.

{% hint style="success" %}
Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

```bash
export LLAMA_CACHE="unsloth/GLM-4.6-GGUF"
./llama.cpp/llama-cli \
    --model GLM-4.6-GGUF/UD-Q2_K_XL/GLM-4.6-UD-Q2_K_XL-00001-of-00003.gguf \
    --n-gpu-layers 99 \
    --jinja \
    --ctx-size 16384 \
    --flash-attn on \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 40 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-`Q2\_K\_XL (dynamic 2bit quant) or other quantized versions like `Q4_K_XL` . We <mark style="background-color:green;">**recommend using our 2.7bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/GLM-4.6-GGUF",
    local_dir = "unsloth/GLM-4.6-GGUF",
    allow_patterns = ["*UD-Q2_K_XL*"], # Dynamic 2bit Use "*UD-TQ1_0*" for Dynamic 1bit
)
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/GLM-4.6-GGUF/UD-Q2_K_XL/GLM-4.6-UD-Q2_K_XL-00001-of-00003.gguf \
    --jinja \
    --n-gpu-layers 99 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 40 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endcode %}
{% endstep %}
{% endstepper %}

### ✨ Deploy with llama-server and OpenAI's completion library

To use llama-server for deployment, use the following command:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/GLM-4.6-GGUF/UD-Q2_K_XL/GLM-4.6-UD-Q2_K_XL-00001-of-00003.gguf \
    --alias "unsloth/GLM-4.6" \
    --n-gpu-layers 999 \
    -ot ".ffn_.*_exps.=CPU" \
    --prio 3 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 40 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

{% endcode %}

Then use OpenAI's Python library after `pip install openai` :

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/GLM-4.6",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
```

### :minidisc:Model uploads

**ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks.

* Full GLM-4.6 model uploads below:

We also uploaded [IQ4\_NL](https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/IQ4_NL) and [Q4\_1](https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/Q4_1) quants which run specifically faster for ARM and Apple devices respectively.

<table data-full-width="false"><thead><tr><th>MoE Bits</th><th>Type + Link</th><th>Disk Size</th><th>Details</th></tr></thead><tbody><tr><td>1.66bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF?show_file_info=GLM-4.6-UD-TQ1_0.gguf">TQ1_0</a></td><td><strong>84GB</strong></td><td>1.92/1.56bit</td></tr><tr><td>1.78bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-IQ1_S">IQ1_S</a></td><td><strong>96GB</strong></td><td>2.06/1.56bit</td></tr><tr><td>1.93bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-IQ1_M">IQ1_M</a></td><td><strong>107GB</strong></td><td>2.5/2.06/1.56</td></tr><tr><td>2.42bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-IQ2_XXS">IQ2_XXS</a></td><td><strong>115GB</strong></td><td>2.5/2.06bit</td></tr><tr><td>2.71bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-Q2_K_XL">Q2_K_XL</a></td><td><strong>135GB</strong></td><td>3.5/2.5bit</td></tr><tr><td>3.12bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-IQ3_XXS">IQ3_XXS</a></td><td><strong>145GB</strong></td><td>3.5/2.06bit</td></tr><tr><td>3.5bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-Q3_K_XL">Q3_K_XL</a></td><td><strong>158GB</strong></td><td>4.5/3.5bit</td></tr><tr><td>4.5bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-Q4_K_XL">Q4_K_XL</a></td><td><strong>204GB</strong></td><td>5.5/4.5bit</td></tr><tr><td>5.5bit</td><td><a href="https://huggingface.co/unsloth/GLM-4.6-GGUF/tree/main/UD-Q5_K_XL">Q5_K_XL</a></td><td><strong>252GB</strong></td><td>6.5/5.5bit</td></tr></tbody></table>

### :snowboarder: Improving generation speed

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.

Llama.cpp also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel). You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster.

### 📐How to fit long context (full 200K)

To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below.

`--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`

You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1`

You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. Then you can use together with `--cache-type-k` :

`--cache-type-v f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`


# Qwen3-Next: Run Locally Guide

Run Qwen3-Next-80B-A3B-Instruct and Thinking versions locally on your device!

Qwen released Qwen3-Next in Sept 2025, which are 80B MoEs with Thinking and Instruct model variants of [Qwen3](/docs/models/tutorials/qwen3-how-to-run-and-fine-tune). With 256K context, Qwen3-Next was designed with a brand new architecture (Hybrid of MoEs & Gated DeltaNet + Gated Attention) that specifically optimizes for fast inference on longer context lengths. Qwen3-Next has 10x faster inference than Qwen3-32B.

<a href="/docs/models/tutorials/qwen3-next#run-qwen3-next-tutorials" class="button secondary">Run Qwen3-Next Instruct</a><a href="/docs/models/tutorials/qwen3-next#thinking-qwen3-next-80b-a3b-thinking" class="button secondary">Run Qwen3-Next Thinking</a>

Qwen3-Next-80B-A3B Dynamic GGUFs: [**Instruct**](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF) **•** [**Thinking**](https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF)

### ⚙️ Usage Guide

{% hint style="success" %}
NEW as of Dec 6, 2025: Unsloth Qwen3-Next now updated with iMatrix for improved performance.

The thinking model uses `temperature = 0.6`, but the instruct model uses `temperature = 0.7`\
The thinking model uses `top_p = 0.95`, but the instruct model uses `top_p = 0.8`
{% endhint %}

To achieve optimal performance, Qwen recommends these settings:

| Instruct:                                                                                                     | Thinking:                                                                                                     |
| ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| <mark style="background-color:blue;">`Temperature = 0.7`</mark>                                               | <mark style="background-color:blue;">`Temperature = 0.6`</mark>                                               |
| `Min_P = 0.00` (llama.cpp's default is 0.1)                                                                   | `Min_P = 0.00` (llama.cpp's default is 0.1)                                                                   |
| `Top_P = 0.80`                                                                                                | `Top_P = 0.95`                                                                                                |
| `TopK = 20`                                                                                                   | `TopK = 20`                                                                                                   |
| `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) | `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) |

**Adequate Output Length**: Use an output length of `32,768` tokens for most queries for the thinking variant, and `16,384` for the instruct variant. You can increase the max output size for the thinking model if necessary.

Chat template for both Thinking (thinking has `<think></think>`) and Instruct is below:

```
<|im_start|>user
Hey there!<|im_end|>
<|im_start|>assistant
What is 1+1?<|im_end|>
<|im_start|>user
2<|im_end|>
<|im_start|>assistant
```

## 📖 Run Qwen3-Next Tutorials

Below are guides for the [Thinking](#thinking-qwen3-next-80b-a3b-thinking) and [Instruct](#instruct-qwen3-next-80b-a3b-instruct) versions of the model.

### Instruct: Qwen3-Next-80B-A3B-Instruct

Given that this is a non thinking model, the model does not generate `<think> </think>` blocks.

#### ⚙️Best Practices

To achieve optimal performance, Qwen recommends the following settings:

* We suggest using `temperature=0.7, top_p=0.8, top_k=20, and min_p=0.0` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions.
* **`temperature = 0.7`**
* `top_k = 20`
* `min_p = 0.00` (llama.cpp's default is 0.1)
* **`top_p = 0.80`**
* `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example.
* Supports up to `262,144` context natively but you can set it to `32,768` tokens for less RAM use

#### :sparkles: Llama.cpp: Run Qwen3-Next-80B-A3B-Instruct Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. You can directly pull from HuggingFace via:

   ```bash
   ./llama.cpp/llama-cli \
       -hf unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF:Q4_K_XL \
       --jinja -ngl 99 --ctx-size 32768 \
       --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --presence-penalty 1.0
   ```
3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD_Q4_K_XL` or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF",
    local_dir = "Qwen3-Next-80B-A3B-Instruct-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

### Thinking: Qwen3-Next-80B-A3B-Thinking

This model supports only thinking mode and a 256K context window natively. The default chat template adds `<think>` automatically, so you may see only a closing `</think>` tag in the output.

#### ⚙️Best Practices

To achieve optimal performance, Qwen recommends the following settings:

* We suggest using `temperature=0.6, top_p=0.95, top_k=20, and min_p=0.0` `presence_penalty` between 0 and 2 if the framework supports to reduce endless repetitions.
* **`temperature = 0.6`**
* `top_k = 20`
* `min_p = 0.00` (llama.cpp's default is 0.1)
* **`top_p = 0.95`**
* `presence_penalty = 0.0 to 2.0` (llama.cpp default turns it off, but to reduce repetitions, you can use this) Try 1.0 for example.
* Supports up to `262,144` context natively but you can set it to `32,768` tokens for less RAM use

#### :sparkles: Llama.cpp: Run Qwen3-Next-80B-A3B-Thinking Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. You can directly pull from Hugging Face via:

   ```bash
   ./llama.cpp/llama-cli \
       -hf unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF:Q4_K_XL \
       --jinja -ngl 99 --ctx-size 32768 \
       --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence-penalty 1.0
   ```
3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD_Q4_K_XL` or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF",
    local_dir = "Qwen3-Next-80B-A3B-Thinking-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

### 🛠️ Improving generation speed <a href="#improving-generation-speed" id="improving-generation-speed"></a>

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.

The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel). You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster. The [next section](#how-to-fit-long-context-256k-to-1m) talks about KV cache quantization.

### 📐How to fit long context <a href="#how-to-fit-long-context-256k-to-1m" id="how-to-fit-long-context-256k-to-1m"></a>

To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below.

`--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`

You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1` So try out `--cache-type-k q4_1`

You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. After installing Flash Attention, you can then use `--cache-type-v q4_1`

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-be34c101c627020c7a6cfb6cd249f2462587d235%2Fqwen33%20mascot.png?alt=media" alt=""><figcaption></figcaption></figure>


# FunctionGemma: How to Run & Fine-tune

Learn how to run and fine-tune FunctionGemma locally on your device and phone.

FunctionGemma is a new 270M parameter model by Google designed for function-calling and fine-tuning. Based on [Gemma 3](/docs/models/tutorials/gemma-3-how-to-run-and-fine-tune) 270M and trained specifically for text-only tool-calling, its small size makes it great to deploy on your own phone.

You can run the full precision model on **550MB RAM** (CPU) and you can now **fine-tune** it locally with Unsloth. Thank you to Google DeepMind for partnering with Unsloth for day-zero support!

<a href="/docs/models/tutorials/functiongemma#run-functiongemma" class="button secondary">Running Tutorial</a><a href="/docs/models/tutorials/functiongemma#fine-tuning-functiongemma" class="button primary">Fine-tuning FunctionGemma</a>

* FunctionGemma GGUF to run: [unsloth/functiongemma-270m-it-GGUF](https://huggingface.co/unsloth/functiongemma-270m-it-GGUF)

**Free Notebooks:**

* Fine-tune to **reason/think before tool calls** using our [FunctionGemma notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\).ipynb)
* Do **multi-turn tool calling** in a free [Multi Turn tool calling notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Multi-Turn-Tool-Calling.ipynb)
* Fine-tune to **enable mobile actions** (calendar, set timer) in our [Mobile Actions notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Mobile-Actions.ipynb)

### ⚙️ Usage Guide

Google recommends these settings for inference:

* `top_k = 64`
* `top_p = 0.95`
* `temperature = 1.0`
* maximum context length = `32,768`&#x20;

The chat template format is found when we use the below:

{% code overflow="wrap" %}

```python
def get_today_date():
    """ Gets today's date """
    return {"today_date": "18 December 2025"}
    
tokenizer.apply_chat_template(
    [
        {"role" : "user", "content" : "what is today's date?"},
    ],
    tools = [get_today_date], add_generation_prompt = True, tokenize = False,
)
```

{% endcode %}

#### FunctionGemma chat template format:

{% hint style="info" %}
FunctionGemma requires the system or **developer message** as `You are a model that can do function calling with the following functions` Unsloth versions have this pre-built in if you forget to pass one, so please use [unsloth/functiongemma-270m-it](https://huggingface.co/unsloth/functiongemma-270m-it)
{% endhint %}

{% code overflow="wrap" lineNumbers="true" %}

```
<bos><start_of_turn>developer\nYou are a model that can do function calling with the following functions<start_function_declaration>declaration:get_today_date{description:<escape>Gets today's date<escape>,parameters:{type:<escape>OBJECT<escape>}}<end_function_declaration><end_of_turn>\n<start_of_turn>user\nwhat is today's date?<end_of_turn>\n<start_of_turn>model\n
```

{% endcode %}

## 🖥️ Run FunctionGemma

See below for a local desktop guide or you can view our Phone Deployment Guide.

#### Llama.cpp Tutorial (GGUF):

Instructions to run in llama.cpp (note we will be using 4-bit to fit most devices):

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
You can directly pull from Hugging Face. Because the model is so small, we'll be using the unquantized full-precision BF16 variant.

```bash
./llama.cpp/llama-cli \
    -hf unsloth/functiongemma-270m-it-GGUF:BF16 \
    --jinja -ngl 99 --ctx-size 32768 \
    --top-k 64 --top-p 0.95 --temp 1.0
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `BF16` or other quantized versions (though it's not recommended to go lower than 4-bit) due to the small model size.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/functiongemma-270m-it-GGUF",
    local_dir = "unsloth/functiongemma-270m-it-GGUF",
    allow_patterns = ["*BF16*"],
)
```

{% endstep %}

{% step %}
Then run the model in conversation mode:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/functiongemma-270m-it-GGUF/functiongemma-270m-it-BF16.gguf \
    --ctx-size 32768 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --top-k 64 \
    --top-p 0.95 \
    --temp 1.0 \
    --jinja
```

{% endcode %}
{% endstep %}
{% endstepper %}

## 📱 Phone Deployment

You can also run and deploy FunctionGemma on your phone due to its small size. We collaborated with PyTorch to create a streamlined workflow using quantization-aware training ([QAT](/docs/blog/quantization-aware-training-qat)) to recover 70% accuracy then deploying them directly to edge devices.

* Deploy FunctionGemma locally to **Pixel 8** and **iPhone 15 Pro** to get **inference speeds of \~50 tokens/s**
* Get privacy first, instant responses and offline capabilities
* Use our [free Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_\(0_6B\)-Phone_Deployment.ipynb) to fine-tune Qwen3 0.6B and export it for phone deployment - just change it to Gemma3, and follow the [Gemma 3 Executorch docs](https://github.com/pytorch/executorch/tree/main/examples/models/gemma3).

{% content-ref url="/pages/KGLcoYbGfxvjuMoQ3HrB" %}
[Run LLMs on your Phone](/docs/basics/inference-and-deployment/deploy-llms-phone)
{% endcontent-ref %}

View our iOS and Android Tutorials for deploying on your phone:

<a href="/docs/basics/inference-and-deployment/deploy-llms-phone#ios-deployment" class="button secondary" data-icon="apple">iOS Tutorial</a><a href="/docs/basics/inference-and-deployment/deploy-llms-phone#android-deployment" class="button secondary" data-icon="android">Android Tutorial</a>

## 🦥 Fine-tuning FunctionGemma

Google noted that **FunctionGemma is intended to be fine-tuned** for your specific function-calling task, including multi-turn use cases. Unsloth now supports fine-tuning of FunctionGemma. We created 2 fine-tuning notebooks, which shows how you can train the model via **full fine-tuning or LoRA for free via a Colab Notebook:**

{% columns %}
{% column %}
[**Reason before Tool Calling Fine-tuning notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\).ipynb)

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb>" %}
{% endcolumn %}

{% column %}
[**Mobile Actions Fine-tuning notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Mobile-Actions.ipynb)

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-Mobile-Actions.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

In the [**Reason before Tool Calling Fine-tuning notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\).ipynb), we will **fine-tune it "think/reason" before function calling**. Chain-of-thought reasoning is becoming increasingly important for improving tool-use capabilities.

FunctionGemma is a small model specialized for function calling. It utilizes its own distinct chat template. When provided with tool definitions and a user prompt, it generates a structured output. We can then parse this output to execute the tool, retrieve the results, and use them to generate the final answer.

| Turn Type                | Content                                                                                                                                                                                                                                                                                        |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Developer Prompt**     | <p><code>\<start\_of\_turn>developer</code></p><p><code>You can do function calling with the following functions:</code></p>                                                                                                                                                                   |
| **Function Declaration** | <p><code>\<start\_function\_declaration>declaration:get\_weather{</code></p><p><code>description: "Get weather for city",</code></p><p><code>parameters: { city: STRING }</code></p><p><code>}</code></p><p><code>\<end\_function\_declaration></code></p><p><code>\<end\_of\_turn></code></p> |
| **User Turn**            | <p><code>\<start\_of\_turn>user</code></p><p><code>What is the weather like in Paris?</code></p><p><code>\<end\_of\_turn></code></p>                                                                                                                                                           |
| **Function Call**        | <p><code>\<start\_of\_turn>model</code></p><p><code>\<start\_function\_call>call:get\_weather{</code></p><p><code>city: "paris"</code></p><p><code>}</code></p><p><code>\<end\_function\_call></code></p>                                                                                      |
| **Function Response**    | <p><code>\<start\_function\_response>response:get\_weather{temperature:26}</code></p><p><code>\<end\_function\_response></code></p>                                                                                                                                                            |
| **Assistant Closing**    | <p><code>The weather in Paris is 26 degrees Celsius.</code></p><p><code>\<end\_of\_turn></code></p>                                                                                                                                                                                            |

Here, we implement a simplified version using a single thinking block (rather than interleaved reasoning) via `<think></think>`. Consequently, our model interaction looks like this:

| **Thinking** + **Function Call** | <p><code>\<start\_of\_turn>model</code></p><p><mark style="color:blue;"><strong><code>\<think></code></strong></mark></p><p><mark style="color:blue;"><strong><code>The user wants weather for Paris. I have the get\_weather tool. I should call it with the city argument.</code></strong></mark></p><p><mark style="color:blue;"><strong><code>\</think></code></strong></mark></p><p><code>\<start\_function\_call>call:get\_weather{</code></p><p><code>city: "paris"</code></p><p><code>}</code></p><p><code>\<end\_function\_call></code></p> |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

## :accordion:Fine-tuning FunctionGemma for Mobile Actions

We also created a notebook to show how you can make FunctionGemma perform mobile actions. In the [**Mobile Actions Fine-tuning notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Mobile-Actions.ipynb), we enabled evaluation as well, and show how finetuning it for on device actions works well, as seen in the evaluation loss doing down:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkRax7Sl7fN7fN3GoFgiG%2Fimage.png?alt=media&amp;token=e9b648f9-27ae-408b-aa0d-0baf77e5403d" alt="" width="375"><figcaption></figcaption></figure>

For example given a prompt `Please set a reminder for a "Team Sync Meeting" this Friday, June 6th, 2025, at 2 PM.`

{% code overflow="wrap" %}

```python
[{'role': 'developer',
  'content': 'Current date and time given in YYYY-MM-DDTHH:MM:SS format: 2025-06-04T15:29:23\nDay of week is Wednesday\nYou are a model that can do function calling with the following functions\n',
  'tool_calls': None},
 {'role': 'user',
  'content': 'Please set a reminder for a "Team Sync Meeting" this Friday, June 6th, 2025, at 2 PM.',
  'tool_calls': None}]
```

{% endcode %}

We fine-tuned the model to be able to output:

{% code overflow="wrap" %}

```
<start_of_turn>user
Please set a reminder for a "Team Sync Meeting" this Friday, June 6th, 2025, at 2 PM.<end_of_turn>
<start_of_turn>model
<start_function_call>call:create_calendar_event{body:None,datetime:2025-06-06 14:00:00,email:None,first_name:None,last_name:None,phone_number:None,query:None,subject:None,title:<escape>Team Sync Meeting<escape>,to:None}<end_function_call><start_function_response>
```

{% endcode %}

## :man\_running:Multi Turn Tool Calling with FunctionGemma

We also created a notebook to show how you can make FunctionGemma do multi turn tool calls. In the [**Multi Turn tool calling notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Multi-Turn-Tool-Calling.ipynb), we show how FunctionGemma is capable of calling tools in a long message change, for example see below:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F5bewasaWZp4r48LSL0fd%2Fimage.png?alt=media&amp;token=6aab6fd4-1560-4a46-ac41-305764019440" alt=""><figcaption></figcaption></figure>

You first have to specify your tools like below:

{% code expandable="true" %}

```python
def get_today_date():
    """
    Gets today's date

    Returns:
        today_date: Today's date in format 18 December 2025
    """
    from datetime import datetime
    today_date = datetime.today().strftime("%d %B %Y")
    return {"today_date": today_date}

def get_current_weather(location: str, unit: str = "celsius"):
    """
    Gets the current weather in a given location.

    Args:
        location: The city and state, e.g. "San Francisco, CA, USA" or "Sydney, Australia"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])

    Returns:
        temperature: The current temperature in the given location
        weather: The current weather in the given location
    """
    if "San Francisco" in location.title():
        return {"temperature": 15, "weather": "sunny"}
    elif "Sydney" in location.title():
        return {"temperature": 25, "weather": "cloudy"}
    else:
        return {"temperature": 30, "weather": "rainy"}

def add_numbers(x: float | str, y: float | str):
    """
    Adds 2 numbers together

    Args:
        x: First number
        y: Second number

    Returns:
        result: x + y
    """
    return {"result" : float(x) + float(y)}

def multiply_numbers(x: float | str, y: float | str):
    """
    Multiplies 2 numbers together

    Args:
        x: First number
        y: Second number

    Returns:
        result: x * y
    """
    return {"result" : float(x) * float(y)}
```

{% endcode %}

We then create a mapping for all the tools:

```python
FUNCTION_MAPPING = {
    "get_today_date" : get_today_date,
    "get_current_weather" : get_current_weather,
    "add_numbers": add_numbers,
    "multiply_numbers": multiply_numbers,
}
TOOLS = list(FUNCTION_MAPPING.values())
```

We also need some tool invocation and parsing code:

{% code expandable="true" %}

```python
#@title FunctionGemma parsing code (expandible)
import re
def extract_tool_calls(text):
    def cast(v):
        try: return int(v)
        except:
            try: return float(v)
            except: return {'true': True, 'false': False}.get(v.lower(), v.strip("'\""))

    return [{
        "name": name,
        "arguments": {
            k: cast((v1 or v2).strip())
            for k, v1, v2 in re.findall(r"(\w+):(?:<escape>(.*?)<escape>|([^,}]*))", args)
        }
    } for name, args in re.findall(r"<start_function_call>call:(\w+)\{(.*?)\}<end_function_call>", text, re.DOTALL)]

def process_tool_calls(output, messages):
    calls = extract_tool_calls(output)
    if not calls: return messages
    messages.append({
        "role": "assistant",
        "tool_calls": [{"type": "function", "function": call} for call in calls]
    })
    results = [
        {"name": c['name'], "response": FUNCTION_MAPPING[c['name']](**c['arguments'])}
        for c in calls
    ]
    messages.append({ "role": "tool", "content": results })
    return messages

def _do_inference(model, messages, max_new_tokens = 128):
    inputs = tokenizer.apply_chat_template(
        messages, tools = TOOLS, add_generation_prompt = True, return_dict = True, return_tensors = "pt",
    )
    output = tokenizer.decode(inputs["input_ids"][0], skip_special_tokens = False)

    out = model.generate(**inputs.to(model.device), max_new_tokens = max_new_tokens,
                         top_p = 0.95, top_k = 64, temperature = 1.0,)
    generated_tokens = out[0][len(inputs["input_ids"][0]):]
    return tokenizer.decode(generated_tokens, skip_special_tokens = True)
    
def do_inference(model, messages, print_assistant = True, max_new_tokens = 128):
    output = _do_inference(model, messages, max_new_tokens = max_new_tokens)
    messages = process_tool_calls(output, messages)
    if messages[-1]["role"] == "tool":
        output = _do_inference(model, messages, max_new_tokens = max_new_tokens)
    messages.append({"role": "assistant", "content": output})
    if print_assistant: print(output)
    return messages
```

{% endcode %}

And now we can call the model!

```python
from unsloth import FastLanguageModel
import torch
max_seq_length = 4096 # Can choose any sequence length!
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/functiongemma-270m-it",
    max_seq_length = max_seq_length, # Choose any for long context!
    load_in_4bit = False,  # 4 bit quantization to reduce memory
    load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory
    load_in_16bit = True, # [NEW!] Enables 16bit LoRA
    full_finetuning = False, # [NEW!] We have full finetuning now!
    # token = "hf_...", # use one if using gated models
)

messages = []
messages.append({"role": "user", "content": "What's today's date?"})
messages = do_inference(model, messages, max_new_tokens = 128)
```

Try the 3 notebooks we made for FunctionGemma:

{% columns %}
{% column %}
[Reason before Tool Calling Fine-tuning notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\).ipynb)

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M).ipynb>" %}
{% endcolumn %}

{% column %}
[Mobile Actions Fine-tuning notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Mobile-Actions.ipynb)

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-Mobile-Actions.ipynb>" %}
{% endcolumn %}

{% column %}
[Multi Turn tool calling notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_\(270M\)-Multi-Turn-Tool-Calling.ipynb)

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/FunctionGemma_(270M)-Multi-Turn-Tool-Calling.ipynb>" %}
{% endcolumn %}
{% endcolumns %}


# DeepSeek-V3.1: How to Run Locally

A guide on how to run DeepSeek-V3.1 and Terminus on your own local device!

DeepSeek’s V3.1 and **Terminus** update introduces hybrid reasoning inference, combining 'think' and 'non-think' into one model. The full 671B parameter model requires 715GB of disk space. The quantized dynamic 2-bit version uses 245GB (-75% reduction in size). GGUF: [**DeepSeek-V3.1-GGUF**](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF)

{% hint style="success" %}
**NEW:** DeepSeek-V3.1-Terminus out now: [DeepSeek-V3.1-Terminus-GGUF](https://huggingface.co/unsloth/DeepSeek-V3.1-Terminus-GGUF)\
\
[**Sept 10, 2025 update:**](/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot) You asked for tougher benchmarks, so we’re showcasing Aider Polyglot results! Our Dynamic 3-bit DeepSeek V3.1 GGUF scores **75.6%**, surpassing many full-precision SOTA LLMs. [Read more.](/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot)

Our DeepSeek-V3.1 GGUFs include Unsloth [chat template fixes](#chat-template-bug-fixes) for llama.cpp supported backends.
{% endhint %}

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized DeepSeek LLMs with minimal accuracy loss.

**Tutorials navigation:**

<a href="#run-in-llama.cpp" class="button secondary">Run in llama.cpp</a><a href="#run-in-ollama-open-webui" class="button secondary">Run in Ollama/Open WebUI</a>

## :gear: Recommended Settings

The 1-bit dynamic quant TQ1\_0 (1bit for unimportant MoE layers, 2-4bit for important MoE, and 6-8bit for rest) uses 170GB of disk space - this works well in a **1x24GB card and 128GB of RAM** with MoE offloading - it also **works natively in Ollama**!

{% hint style="info" %}
You must use `--jinja` for llama.cpp quants - this uses our [fixed chat templates](#chat-template-bug-fixes) and enables the correct template! You might get incorrect results if you do not use `--jinja`
{% endhint %}

The 2-bit quants will fit in a 1x 24GB GPU (with MoE layers offloaded to RAM). Expect around 5 tokens/s with this setup if you have bonus 128GB RAM as well. It is recommended to have at least 226GB RAM to run this 2-bit. For optimal performance you will need at least 226GB unified memory or 226GB combined RAM+VRAM for 5+ tokens/s. To learn how to increase generation speed and fit longer contexts, [read here](#improving-generation-speed).

{% hint style="success" %}
Though not a must, for best performance, have your VRAM + RAM combined equal to the size of the quant you're downloading. If not, hard drive / SSD offloading will work with llama.cpp, just inference will be slower.
{% endhint %}

## :butterfly:Chat template bug fixes

We fixed a few issues with DeepSeek V3.1's chat template since they did not function correctly in llama.cpp and other engines:

1. DeepSeek V3.1 is a hybrid reasoning model, meaning you can change the chat template to enable reasoning. The chat template introduced `thinking = True` , but other models use `enable_thinking = True` . We added the option to use `enable_thinking` as a keyword instead.
2. llama.cpp's jinja renderer via [minja](https://github.com/google/minja) does not allow the use of extra arguments in the `.split()` command, so using `.split(text, 1)` works in Python, but not in minja. We had to change this to make llama.cpp function correctly without erroring out.\
   \
   You will get the following error when using other quants:\
   `terminate called after throwing an instance of 'std::runtime_error' what(): split method must have between 1 and 1 positional arguments and between 0 and 0 keyword arguments at row 3, column 1908` We fixed it in all our quants!

### 🐳Official Recommended Settings

According to [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-V3.1), these are the recommended settings for V3.1 inference:

* Set the <mark style="background-color:green;">**temperature 0.6**</mark> to reduce repetition and incoherence.
* Set <mark style="background-color:green;">**top\_p to 0.95**</mark> (recommended)
* **128K context length** or less
* Use `--jinja` for llama.cpp variants - we **fixed some chat template issues as well!**
* **Use** `enable_thinking = True` to use reasoning/ thinking mode. By default it's set to non reasoning.

#### :1234: Chat template/prompt format

You do not need to force `<think>\n` , but you can still add it in! With the given prefix, DeepSeek V3.1 generates responses to queries in non-thinking mode. Unlike DeepSeek V3, it introduces an additional token `</think>`.

```
<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜></think>
```

A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call `tokenizer.encode(..., add_special_tokens = False)` since the chat template auto adds a BOS token as well. For llama.cpp / GGUF inference, you should skip the BOS since it’ll auto add it.

#### :notebook\_with\_decorative\_cover: Non-Thinking Mode (use `thinking = False`or `enable_thinking = False` and is by default)

**First-Turn**

Prefix: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜></think>`

With the given prefix, DeepSeek V3.1 generates responses to queries in non-thinking mode. Unlike DeepSeek V3, it introduces an additional token `</think>`.

**Multi-Turn**

Context: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>...<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>`

Prefix: `<｜User｜>{query}<｜Assistant｜></think>`

By concatenating the context and the prefix, we obtain the correct prompt for the query.

#### :books: Thinking Mode (use `thinking = True`or `enable_thinking = True` and is by default)

**First-Turn**

Prefix: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜><think>`

The prefix of thinking mode is similar to DeepSeek-R1.

**Multi-Turn**

Context: `<｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>...<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>`

Prefix: `<｜User｜>{query}<｜Assistant｜><think>`

The multi-turn template is the same with non-thinking multi-turn chat template. It means the thinking token in the last turn will be dropped but the `</think>` is retained in every turn of context.

#### :bow\_and\_arrow: Tool Calling

Tool calling is supported in non-thinking mode. The format is:

`<｜begin▁of▁sentence｜>{system prompt}{tool_description}<｜User｜>{query}<｜Assistant｜></think>` where we populate the tool\_description is area after the system prompt.

## :arrow\_forward:Run DeepSeek-V3.1 Tutorials:

### :llama: Run in Ollama/Open WebUI

{% stepper %}
{% step %}
Install `ollama` if you haven't already! To run more variants of the model, [see here](#run-in-llama.cpp).

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

{% endstep %}

{% step %}
Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!\
**(NEW) To DeepSeek-V3.1 model in Ollama, you can use our TQ1\_0 (170GB quant):**

```bash
OLLAMA_MODELS=unsloth ollama serve &

OLLAMA_MODELS=unsloth ollama run hf.co/unsloth/DeepSeek-V3.1-Terminus-GGUF:TQ1_0
```

{% endstep %}

{% step %}
To run other quants, you need to first merge the GGUF split files into 1 like the code below. Then you will need to run the model locally.

```bash
./llama.cpp/llama-gguf-split --merge \
  DeepSeek-V3.1-Terminus-GGUF/DeepSeek-V3.1-Terminus-UD-Q2_K_XL/DeepSeek-V3.1-Terminus-UD-Q2_K_XL-00001-of-00006.gguf \
	merged_file.gguf
```

```bash
OLLAMA_MODELS=unsloth ollama serve &

OLLAMA_MODELS=unsloth ollama run merged_file.gguf
```

{% endstep %}

{% step %}
Open WebUI also made a [step-by-step tutorial](https://docs.openwebui.com/tutorials/integrations/deepseekr1-dynamic/) on how to run R1 and for V3.1, you will just need to replace R1 with the new V3.1 quant.
{% endstep %}
{% endstepper %}

### ✨ Run in llama.cpp

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:Q2\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 128K context length.

{% hint style="success" %}
Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

```bash
export LLAMA_CACHE="unsloth/DeepSeek-V3.1-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q2_K_XL \
    --jinja \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-`Q2\_K\_XL (dynamic 2bit quant) or other quantized versions like `Q4_K_M` . We <mark style="background-color:green;">**recommend using our 2.7bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/DeepSeek-V3.1-Terminus-GGUF",
    local_dir = "unsloth/DeepSeek-V3.1-Terminus-GGUF",
    allow_patterns = ["*UD-Q2_K_XL*"], # Dynamic 2bit Use "*UD-TQ1_0*" for Dynamic 1bit
)
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-V3.1-Terminus-GGUF/UD-Q2_K_XL/DeepSeek-V3.1-Terminus-UD-Q2_K_XL-00001-of-00006.gguf \
    --jinja \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endcode %}
{% endstep %}

{% step %}
Get the 1bit version (170GB) if you don't have enough combined RAM and VRAM:

```python
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/DeepSeek-V3.1-Terminus-GGUF",
    local_dir = "unsloth/DeepSeek-V3.1-Terminus-GGUF",
    allow_patterns = ["*UD-TQ1_0*"], # Use "*UD-Q2_K_XL*" for Dynamic 2bit
)
```

{% endstep %}
{% endstepper %}

### ✨ Deploy with llama-server and OpenAI's completion library

To use llama-server for deployment, use the following command:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model unsloth/DeepSeek-V3.1-Terminus-GGUF/DeepSeek-V3.1-Terminus-UD-TQ1_0.gguf \
    --alias "unsloth/DeepSeek-V3.1-Terminus" \
    --n-gpu-layers 999 \
    -ot ".ffn_.*_exps.=CPU" \
    --prio 3 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

{% endcode %}

Then use OpenAI's Python library after `pip install openai` :

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/DeepSeek-V3.1-Terminus",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
```

## :minidisc:Model uploads

**ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks.

* Full DeepSeek-V3.1 model uploads below:

We also uploaded [IQ4\_NL](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/IQ4_NL) and [Q4\_1](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/Q4_1) quants which run specifically faster for ARM and Apple devices respectively.

<table data-full-width="false"><thead><tr><th>MoE Bits</th><th>Type + Link</th><th>Disk Size</th><th>Details</th></tr></thead><tbody><tr><td>1.66bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF?show_file_info=DeepSeek-V3.1-UD-TQ1_0.gguf">TQ1_0</a></td><td><strong>170GB</strong></td><td>1.92/1.56bit</td></tr><tr><td>1.78bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ1_S">IQ1_S</a></td><td><strong>185GB</strong></td><td>2.06/1.56bit</td></tr><tr><td>1.93bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ1_M">IQ1_M</a></td><td><strong>200GB</strong></td><td>2.5/2.06/1.56</td></tr><tr><td>2.42bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ2_XXS">IQ2_XXS</a></td><td><strong>216GB</strong></td><td>2.5/2.06bit</td></tr><tr><td>2.71bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q2_K_XL">Q2_K_XL</a></td><td><strong>251GB</strong></td><td>3.5/2.5bit</td></tr><tr><td>3.12bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-IQ3_XXS">IQ3_XXS</a></td><td><strong>273GB</strong></td><td>3.5/2.06bit</td></tr><tr><td>3.5bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q3_K_XL">Q3_K_XL</a></td><td><strong>296GB</strong></td><td>4.5/3.5bit</td></tr><tr><td>4.5bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q4_K_XL">Q4_K_XL</a></td><td><strong>384GB</strong></td><td>5.5/4.5bit</td></tr><tr><td>5.5bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF/tree/main/UD-Q5_K_XL">Q5_K_XL</a></td><td><strong>481GB</strong></td><td>6.5/5.5bit</td></tr></tbody></table>

We've also uploaded versions in [BF16 format](https://huggingface.co/unsloth/DeepSeek-V3.1-BF16), and original [FP8 (float8) format](https://huggingface.co/unsloth/DeepSeek-V3.1).

## :snowboarder: Improving generation speed

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.

The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel). You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster.

## 📐How to fit long context (full 128K)

To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below.

`--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`

You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1`

You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. Then you can use together with `--cache-type-k` :

`--cache-type-v f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`


# DeepSeek-R1-0528: How to Run Locally

A guide on how to run DeepSeek-R1-0528 including Qwen3 on your own local device!

DeepSeek-R1-0528 is DeepSeek's new update to their R1 reasoning model. The full 671B parameter model requires 715GB of disk space. The quantized dynamic **1.66-bit** version uses 162GB (-80% reduction in size). GGUF: [DeepSeek-R1-0528-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF)

DeepSeek also released a R1-0528 distilled version by fine-tuning Qwen3 (8B). The distill achieves similar performance to Qwen3 (235B). ***You can also*** [***fine-tune Qwen3 Distill***](#fine-tuning-deepseek-r1-0528-with-unsloth) ***with Unsloth***. Qwen3 GGUF: [DeepSeek-R1-0528-Qwen3-8B-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF)

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized DeepSeek LLMs with minimal accuracy loss.

**Tutorials navigation:**

<a href="#run-qwen3-distilled-r1-in-llama.cpp" class="button secondary">Run in llama.cpp</a><a href="#run-in-ollama-open-webui" class="button secondary">Run in Ollama/Open WebUI</a><a href="#fine-tuning-deepseek-r1-0528-with-unsloth" class="button secondary">Fine-tuning R1-0528</a>

{% hint style="success" %}
NEW: Huge improvements to tool calling and chat template fixes.\
\
New [TQ1\_0 dynamic 1.66-bit quant](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF?show_file_info=DeepSeek-R1-0528-UD-TQ1_0.gguf) - 162GB in size. Ideal for 192GB RAM (including Mac) and Ollama users. Try: `ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0`
{% endhint %}

## :gear: Recommended Settings

For DeepSeek-R1-0528-Qwen3-8B, the model can pretty much fit in any setup, and even those with as less as 20GB RAM. There is no need for any prep beforehand.\
\
However, for the full R1-0528 model which is 715GB in size, you will need extra prep. The 1.78-bit (IQ1\_S) quant will fit in a 1x 24GB GPU (with all layers offloaded). Expect around 5 tokens/s with this setup if you have bonus 128GB RAM as well.

It is recommended to have at least 64GB RAM to run this quant (you will get 1 token/s without a GPU). For optimal performance you will need at least **180GB unified memory or 180GB combined RAM+VRAM** for 5+ tokens/s.

We suggest using our 2.7bit (Q2\_K\_XL) or 2.4bit (IQ2\_XXS) quant to balance size and accuracy! The 2.4bit one also works well.

{% hint style="success" %}
Though not necessary, for the best performance, have your VRAM + RAM combined = to the size of the quant you're downloading.
{% endhint %}

### 🐳 Official Recommended Settings:

According to [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528), these are the recommended settings for R1 (R1-0528 and Qwen3 distill should use the same settings) inference:

* Set the <mark style="background-color:green;">**temperature 0.6**</mark> to reduce repetition and incoherence.
* Set <mark style="background-color:green;">**top\_p to 0.95**</mark> (recommended)
* Run multiple tests and average results for reliable evaluation.

### :1234: Chat template/prompt format

R1-0528 uses the same chat template as the original R1 model. You do not need to force `<think>\n` , but you can still add it in!

```
<｜begin▁of▁sentence｜><｜User｜>What is 1+1?<｜Assistant｜>It's 2.<｜end▁of▁sentence｜><｜User｜>Explain more!<｜Assistant｜>
```

A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call `tokenizer.encode(..., add_special_tokens = False)` since the chat template auto adds a BOS token as well.\
For llama.cpp / GGUF inference, you should skip the BOS since it’ll auto add it:

```
<｜User｜>What is 1+1?<｜Assistant｜>
```

The `<think>` and `</think>` tokens get their own designated tokens.

## Model uploads

**ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks.

* Qwen3 (8B) distill: [DeepSeek-R1-0528-Qwen3-8B-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF)
* Full DeepSeek-R1-0528 model uploads below:

We also uploaded [IQ4\_NL](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/IQ4_NL) and [Q4\_1](https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/Q4_1) quants which run specifically faster for ARM and Apple devices respectively.

<table data-full-width="false"><thead><tr><th>MoE Bits</th><th>Type + Link</th><th>Disk Size</th><th>Details</th></tr></thead><tbody><tr><td>1.66bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF?show_file_info=DeepSeek-R1-0528-UD-TQ1_0.gguf">TQ1_0</a></td><td><strong>162GB</strong></td><td>1.92/1.56bit</td></tr><tr><td>1.78bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ1_S">IQ1_S</a></td><td><strong>185GB</strong></td><td>2.06/1.56bit</td></tr><tr><td>1.93bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ1_M">IQ1_M</a></td><td><strong>200GB</strong></td><td>2.5/2.06/1.56</td></tr><tr><td>2.42bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ2_XXS">IQ2_XXS</a></td><td><strong>216GB</strong></td><td>2.5/2.06bit</td></tr><tr><td>2.71bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q2_K_XL">Q2_K_XL</a></td><td><strong>251GB</strong></td><td>3.5/2.5bit</td></tr><tr><td>3.12bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-IQ3_XXS">IQ3_XXS</a></td><td><strong>273GB</strong></td><td>3.5/2.06bit</td></tr><tr><td>3.5bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q3_K_XL">Q3_K_XL</a></td><td><strong>296GB</strong></td><td>4.5/3.5bit</td></tr><tr><td>4.5bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q4_K_XL">Q4_K_XL</a></td><td><strong>384GB</strong></td><td>5.5/4.5bit</td></tr><tr><td>5.5bit</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF/tree/main/UD-Q5_K_XL">Q5_K_XL</a></td><td><strong>481GB</strong></td><td>6.5/5.5bit</td></tr></tbody></table>

We've also uploaded versions in [BF16 format](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16), and original [FP8 (float8) format](https://huggingface.co/unsloth/DeepSeek-R1-0528).

## Run DeepSeek-R1-0528 Tutorials:

### :llama: Run in Ollama/Open WebUI

1. Install `ollama` if you haven't already! You can only run models up to 32B in size. To run the full 720GB R1-0528 model, [see here](#run-full-r1-0528-on-ollama-open-webui).

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload!

```bash
ollama run hf.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q4_K_XL
```

3. <mark style="color:green;background-color:yellow;">**(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1\_0 (162GB quant):**</mark>

```bash
OLLAMA_MODELS=unsloth_downloaded_models ollama serve &

ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0
```

### :llama: Run Full R1-0528 on Ollama/Open WebUI

Open WebUI has made an step-by-step tutorial on how to run R1 here and for R1-0528, you will just need to replace R1 with the new 0528 quant: <https://docs.openwebui.com/tutorials/integrations/llm-providers/deepseekr1-dynamic>

<mark style="background-color:green;">**(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1\_0 (162GB quant):**</mark>

```bash
OLLAMA_MODELS=unsloth_downloaded_models ollama serve &

ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0
```

If you want to use any of the quants that are larger than TQ1\_0 (162GB) on Ollama, you need to first merge the 3 GGUF split files into 1 like the code below. Then you will need to run the model locally.

```bash
./llama.cpp/llama-gguf-split --merge \
  DeepSeek-R1-0528-GGUF/DeepSeek-R1-0528-UD-IQ1_S/DeepSeek-R1-0528-UD-IQ1_S-00001-of-00003.gguf \
	merged_file.gguf
```

### ✨ Run Qwen3 distilled R1 in llama.cpp

1. <mark style="background-color:yellow;">**To run the full 720GB R1-0528 model,**</mark> [<mark style="background-color:yellow;">**see here**</mark>](#run-full-r1-0528-on-llama.cpp)<mark style="background-color:yellow;">**.**</mark> Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. Then use llama.cpp directly to download the model:

```bash
./llama.cpp/llama-cli -hf unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q4_K_XL --jinja
```

### ✨ Run Full R1-0528 on llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:IQ1\_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location.

{% hint style="success" %}
Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

```bash
export LLAMA_CACHE="unsloth/DeepSeek-R1-0528-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/DeepSeek-R1-0528-GGUF:IQ1_S \
    --cache-type-k q4_0 \
    --threads -1 \
    --n-gpu-layers 99 \
    --prio 3 \
    --temp 0.6 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-IQ1_S`(dynamic 1.78bit quant) or other quantized versions like `Q4_K_M` . We <mark style="background-color:green;">**recommend using our 2.7bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>. More versions at: <https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF>

{% code overflow="wrap" %}

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/DeepSeek-R1-0528-GGUF",
    local_dir = "unsloth/DeepSeek-R1-0528-GGUF",
    allow_patterns = ["*UD-IQ1_S*"], # Dynamic 1bit (168GB) Use "*UD-Q2_K_XL*" for Dynamic 2bit (251GB)
)
```

{% endcode %}

4. Run Unsloth's Flappy Bird test as described in our 1.58bit Dynamic Quant for DeepSeek R1.
5. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-R1-0528-GGUF/UD-IQ1_S/DeepSeek-R1-0528-UD-IQ1_S-00001-of-00004.gguf \
    --cache-type-k q4_0 \
    --threads -1 \
    --n-gpu-layers 99 \
    --prio 3 \
    --temp 0.6 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU" \
    -no-cnv \
    --prompt "<｜User｜>Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<｜Assistant｜>"
```

{% endcode %}

## :8ball: Heptagon Test

You can also test our dynamic quants via [r/Localllama](https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/) which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1371de5a93c6c5b0e43e8bb51980d84554b199f4%2Fsnapshot.jpg?alt=media" alt="" width="563"><figcaption><p>The goal is to make the heptagon spin, and the balls in the heptagon should move.</p></figcaption></figure>

<details>

<summary>Full prompt to run the model</summary>

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-R1-0528-GGUF/UD-IQ1_S/DeepSeek-R1-0528-UD-IQ1_S-00001-of-00004.gguf \
    --cache-type-k q4_0 \
    --threads -1 \
    --n-gpu-layers 99 \
    --prio 3 \
    --temp 0.6 \
    --top_p 0.95 \
    --min_p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU" \
    -no-cnv \
    --prompt "<｜User｜>Write a Python program that shows 20 balls bouncing inside a spinning heptagon:\n- All balls have the same radius.\n- All balls have a number on it from 1 to 20.\n- All balls drop from the heptagon center when starting.\n- Colors are: #f8b862, #f6ad49, #f39800, #f08300, #ec6d51, #ee7948, #ed6d3d, #ec6800, #ec6800, #ee7800, #eb6238, #ea5506, #ea5506, #eb6101, #e49e61, #e45e32, #e17b34, #dd7a56, #db8449, #d66a35\n- The balls should be affected by gravity and friction, and they must bounce off the rotating walls realistically. There should also be collisions between balls.\n- The material of all the balls determines that their impact bounce height will not exceed the radius of the heptagon, but higher than ball radius.\n- All balls rotate with friction, the numbers on the ball can be used to indicate the spin of the ball.\n- The heptagon is spinning around its center, and the speed of spinning is 360 degrees per 5 seconds.\n- The heptagon size should be large enough to contain all the balls.\n- Do not use the pygame library; implement collision detection algorithms and collision response etc. by yourself. The following Python libraries are allowed: tkinter, math, numpy, dataclasses, typing, sys.\n- All codes should be put in a single Python file.<｜Assistant｜>"
```

{% endcode %}

</details>

## 🦥 Fine-tuning DeepSeek-R1-0528 with Unsloth

To fine-tune **DeepSeek-R1-0528-Qwen3-8B** using Unsloth, we’ve made a new GRPO notebook featuring a custom reward function designed to significantly enhance multilingual output - specifically increasing the rate of desired language responses (in our example we use Indonesian but you can use any) by more than 40%.

* [**DeepSeek-R1-0528-Qwen3-8B notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/DeepSeek_R1_0528_Qwen3_\(8B\)_GRPO.ipynb) **- new**

While many reasoning LLMs have multilingual capabilities, they often produce mixed-language outputs in its reasoning traces, combining English with the target language. Our reward function effectively mitigates this issue by strongly encouraging outputs in the desired language, leading to a substantial improvement in language consistency.

This reward function is also fully customizable, allowing you to adapt it for other languages or fine-tune for specific domains or use cases.

{% hint style="success" %}
The best part about this whole reward function and notebook is you DO NOT need a language dataset to force your model to learn a specific language. The notebook has no Indonesian dataset.
{% endhint %}

Unsloth makes R1-Qwen3 distill fine-tuning 2× faster, uses 70% less VRAM, and support 8× longer context lengths.


# Liquid LFM2.5: How To Run & Fine-tune

Run and fine-tune LFM2.5 Instruct and Vision locally on your device!

Liquid AI releases LFM2.5, including their [instruct](#run-lfm2.5-1.2b-instruct) and [vision](#liquid-lfm2.5-1.2b-vl-guide) model. LFM2.5-1.2B-Instruct is a 1.17B parameter hybrid reasoning model trained on **28T tokens** and RL, delivering best-in-class performance at the 1B scale for instruction following, tool use, and agentic tasks. See [Hugging Face Jobs](/docs/basics/inference-and-deployment/deploying-llms-with-hugging-face-jobs) on using Codex to train LFM!

LFM2.5 runs on under **1GB RAM** and achieves **239 tok/s** decode on AMD CPU. You can also [**fine-tune** it locally](#fine-tuning-lfm2.5-with-unsloth) with Unsloth.

<a href="/pages/CixcWjfA7MpVVdwbzTmg#run-lfm2.5-1.2b-instruct" class="button primary">Text LFM2.5-Instruct</a><a href="/pages/CixcWjfA7MpVVdwbzTmg#liquid-lfm2.5-1.2b-vl-guide" class="button primary">Vision LFM2.5-VL</a>

| Dynamic GGUFs                                                                         | 16-bit Instruct                                                             |
| ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| [LFM2.5-1.2B-Instruct-GGUF](https://huggingface.co/unsloth/LFM2.5-1.2B-Instruct-GGUF) | [LFM2.5-1.2B-Instruct](https://huggingface.co/unsloth/LFM2.5-1.2B-Instruct) |

**Model Specifications:**

* **Parameters**: 1.17B
* **Architecture**: 16 layers (10 double-gated LIV convolution blocks + 6 GQA blocks)
* **Training Budget**: 28T tokens
* **Context Length**: 32,768 tokens
* **Vocabulary Size**: 65,536
* **Languages**: English, Arabic, Chinese, French, German, Japanese, Korean, Spanish

### ⚙️ Usage Guide

Liquid AI recommends these settings for inference:

* `temperature = 0.1`
* `top_k = 50`
* `top_p = 0.1`
* `repetition_penalty = 1.05`
* Maximum context length: `32,768`

#### Chat Template Format

LFM2.5 uses a ChatML-like format:

```python
tokenizer.apply_chat_template([
    {"role": "system", "content": "You are a helpful assistant trained by Liquid AI."},
    {"role": "user", "content": "What is C. elegans?"},
], add_generation_prompt=True, tokenize=False)
```

**LFM2.5 chat template:**

```
<|startoftext|><|im_start|>system
You are a helpful assistant trained by Liquid AI.<|im_end|>
<|im_start|>user
What is C. elegans?<|im_end|>
<|im_start|>assistant
```

#### Tool Use

LFM2.5 supports function calling with special tokens `<|tool_call_start|>` and `<|tool_call_end|>`. Provide tools as a JSON object in the system prompt:

```
<|startoftext|><|im_start|>system
List of tools: [{"name": "get_weather", "description": "Gets the current weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}]<|im_end|>
<|im_start|>user
What's the weather in Paris?<|im_end|>
<|im_start|>assistant
<|tool_call_start|>[get_weather(city="Paris")]<|tool_call_end|>
```

### 🖥️ Run LFM2.5-1.2B-Instruct

#### 📖 llama.cpp Tutorial (GGUF)

**1. Build llama.cpp**

Obtain the latest `llama.cpp` from [GitHub](https://github.com/ggml-org/llama.cpp). Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
```

**2. Run directly from Hugging Face**

```bash
./llama.cpp/llama-cli \
    -hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF:Q4_K_M \
    --jinja --ctx-size 32768 \
    --temp 0.1 --top-k 50 --top-p 0.1 --repeat-penalty 1.05
```

**3. Or download the model first**

```python
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="LiquidAI/LFM2.5-1.2B-Instruct-GGUF",
    local_dir="LiquidAI/LFM2.5-1.2B-Instruct-GGUF",
    allow_patterns=["*Q4_K_M*"],
)
```

**4. Run in conversation mode**

```bash
./llama.cpp/llama-cli \
    --model LiquidAI/LFM2.5-1.2B-Instruct-GGUF/LFM2.5-1.2B-Instruct-Q4_K_M.gguf \
    --ctx-size 32768 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.1 \
    --top-k 50 \
    --top-p 0.1 \
    --repeat-penalty 1.05 \
    --jinja
```

### 🦥 Fine-tuning LFM2.5 with Unsloth

Unsloth supports fine-tuning LFM2.5 models. The 1.2B model fits comfortably on a free Colab T4 GPU. Training is 2x faster with 50% less VRAM.

**Free Colab Notebook:**

* [LFM2.5-1.2B-Instruct SFT LoRA notebook](https://colab.research.google.com/drive/1vGRg4ksRj__6OLvXkHhvji_Pamv801Ss?usp=sharing)
* [LFM2.5-1.2B-Instruct GRPO LoRA notebook](https://colab.research.google.com/drive/1mIikXFaGvcW4vXOZXLbVTxfBRw_XsXa5?usp=sharing)
* [LFM2.5-1.2B-Base Continued Pretraining (text completion) notebook](https://colab.research.google.com/drive/10fm7eNMezs-DSn36mF7vAsNYlOsx9YZO?usp=sharing)
* [LFM2.5-1.2B-Base Continued Pretraining (translation) notebook](https://colab.research.google.com/drive/1gaP8yTle2_v35Um8Gpu9239fqbU7UgY8?usp=sharing)

LFM2.5 is recommended for agentic tasks, data extraction, RAG, and tool use. It is not recommended for knowledge-intensive tasks or programming.

#### Unsloth Config for LFM2.5

```python
from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="LiquidAI/LFM2.5-1.2B-Instruct",
    max_seq_length=4096,
    load_in_4bit=False,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules = ["q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
                      "w1", "w2", "w3"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)
```

#### Training Setup

```python
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=4096,
    dataset_num_proc=2,
    packing=False,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
    ),
)

trainer.train()
```

#### Save and Export

```python
# Save LoRA adapters
model.save_pretrained("lfm25_lora")
tokenizer.save_pretrained("lfm25_lora")

# Merge and save to 16bit
model.save_pretrained_merged("lfm25_merged", tokenizer, save_method="merged_16bit")

# Export to GGUF
model.save_pretrained_gguf("lfm25_gguf", tokenizer, quantization_method="q4_k_m")
```

### 🎉 llama-server Serving & Deployment

To deploy LFM2.5 for production with an OpenAI-compatible API:

```bash
./llama.cpp/llama-server \
    --model LiquidAI/LFM2.5-1.2B-Instruct-GGUF/LFM2.5-1.2B-Instruct-Q4_K_M.gguf \
    --alias "LiquidAI/LFM2.5-1.2B-Instruct" \
    --threads -1 \
    --n-gpu-layers 99 \
    --ctx-size 32768 \
    --port 8001 \
    --temp 0.1 \
    --top-k 50 \
    --top-p 0.1 \
    --repeat-penalty 1.05 \
    --jinja
```

**Test with OpenAI client:**

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8001/v1",
    api_key="sk-no-key-required",
)

completion = client.chat.completions.create(
    model="LiquidAI/LFM2.5-1.2B-Instruct",
    messages=[{"role": "user", "content": "What is 2+2?"}],
)
print(completion.choices[0].message.content)
```

### 📊 Benchmarks

LFM2.5-1.2B-Instruct delivers best-in-class performance at the 1B scale and offers fast CPU inference with low memory usage:

![](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/dxnYF2fuLpulismtFSGFi.png) ![](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/dbbI-15p9re2ROhAkqnZm.png)

## 💧 Liquid LFM2.5-1.2B-VL Guide

LFM2.5-VL-1.6B is a vision LLM built on top of [LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base) and tuned for stronger real-world performance. You can now **fine-tune** it locally with Unsloth.

<a href="/pages/CixcWjfA7MpVVdwbzTmg#run-lfm2.5-vl-1.6b" class="button primary">Running Tutorial</a><a href="/pages/CixcWjfA7MpVVdwbzTmg#fine-tuning-lfm2.5-with-unsloth-1" class="button primary">Fine-tuning Tutorial</a>

| Dynamic GGUFs                                                             | 16-bit Instruct                                                 |
| ------------------------------------------------------------------------- | --------------------------------------------------------------- |
| [LFM2.5-VL-1.6B-GGUF](https://huggingface.co/unsloth/LFM2.5-VL-1.6B-GGUF) | [LFM2.5-VL-1.6B](https://huggingface.co/unsloth/LFM2.5-VL-1.6B) |

**Model Specifications:**

* **LM Backbone**: LFM2.5-1.2B-Base
* **Vision encoder**: SigLIP2 NaFlex shape-optimized 400M
* **Context length**: 32,768 tokens
* **Vocabulary size**: 65,536
* **Languages**: English, Arabic, Chinese, French, German, Japanese, Korean, and Spanish
* **Native resolution processing**: Handles images up to 512×512 pixels without upscaling and preserves non-standard aspect ratios without distortion
* **Tiling strategy**: Splits large images into non-overlapping 512×512 patches and includes thumbnail encoding for global context
* **Inference-time flexibility**: User-tunable maximum image tokens and tile count for speed/quality tradeoff without retraining

### :gear: Usage Guide

Liquid AI recommends these settings for inference:

* **Text**: `temperature=0.1`, `min_p=0.15`, `repetition_penalty=1.05`
* **Vision**: min\_image\_tokens=64, max\_image\_tokens=256, do\_image\_splitting=True

#### Chat Template Format

LFM2.5-VL uses a ChatML-like format:

```python
tokenizer.apply_chat_template([
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What's in this image?"}
        ]
    },
    {"role": "assistant", "content": "I can see a cat sitting on a couch."}
], tokenize=False)
```

**LFM2.5-VL chat template:**

```
<|startoftext|><|im_start|>system
You are a helpful multimodal assistant by Liquid AI.<|im_end|>
<|im_start|>user
<image>Describe this image.<|im_end|>
<|im_start|>assistant
This image shows a Caenorhabditis elegans (C. elegans) nematode.<|im_end|>
```

### 🖥️  Run LFM2.5-VL-1.6B

#### :book: llama.cpp Tutorial (GGUF)

**1. Build llama.cpp**

Obtain the latest llama.cpp from [GitHub](https://github.com/ggml-org/llama.cpp). Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
```

**2. Run directly from Hugging Face**

```bash
./llama.cpp/llama-cli \
  -hf LiquidAI/LFM2.5-VL-1.6B-GGUF:Q4_0 \
  --image test_image.jpg \
  --image-max-tokens 64 \
  -p "What's in this image?" \
  -n 128
```

### :sloth: Fine-tuning LFM2.5-VL with Unsloth

Unsloth supports fine-tuning LFM2.5 models. The 1.6B model fits comfortably on a free Colab T4 GPU. Training is 2x faster with 50% less VRAM.

**Free Colab Notebook:**

* [LFM2.5-VL-1.6B SFT LoRA notebook](https://colab.research.google.com/drive/1FaR2HSe91YDe88TG97-JVxMygl-rL6vB?usp=sharing)

#### Unsloth Config for LFM2.5

```python
from unsloth import FastVisionModel
import torch

model, tokenizer = FastVisionModel.from_pretrained(
    model_name = "LiquidAI/LFM2.5-VL-1.6B",
    max_seq_length = 4096, 
    load_in_4bit = False, 
)

model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers     = False, # Set to False for now
    finetune_language_layers   = True, # False if not finetuning language layers
    finetune_attention_modules = True, # False if not finetuning attention layers
    finetune_mlp_modules       = True, # False if not finetuning MLP layers
    r = 16,         
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
)
```

#### Training Setup

```python
from unsloth.trainer import UnslothVisionDataCollator
from trl import SFTTrainer, SFTConfig

FastVisionModel.for_training(model) # Enable for training!

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    data_collator = UnslothVisionDataCollator(model, tokenizer), # Must use!
    train_dataset = converted_dataset,
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 30,# num_train_epochs = 1, # Set this instead of max_steps for full train
        learning_rate = 2e-4,
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.001,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
        report_to = "none",     # For Weights and Biases
        remove_unused_columns = False,
        dataset_text_field = "",
        dataset_kwargs = {"skip_prepare_dataset": True},
        max_length = 2048,
    ),
)

trainer.train()
```

#### Save and Export

```python
# Save LoRA adapters
model.save_pretrained("lfm25_lora")
tokenizer.save_pretrained("lfm25_lora")

# Merge and save to 16bit
model.save_pretrained_merged("lfm25_merged", tokenizer, save_method="merged_16bit")

# Export to GGUF
model.save_pretrained_gguf("lfm25_gguf", tokenizer, quantization_method="q4_k_m")
```

### :bar\_chart: Benchmarks

LFM2.5-VL-1.6B delivers best-in-class performance:

| Model              | MMStar | MM-IFEval | BLINK | InfoVQA (Val) | OCRBench (v2) | RealWorldQA | MMMU (Val) | MMMB (avg) | Multilingual MMBench (avg) |
| ------------------ | ------ | --------- | ----- | ------------- | ------------- | ----------- | ---------- | ---------- | -------------------------- |
| **LFM2.5-VL-1.6B** | 50.67  | 52.29     | 48.82 | 62.71         | 41.44         | 64.84       | 40.56      | 76.96      | 65.90                      |
| LFM2-VL-1.6B       | 49.87  | 46.35     | 44.50 | 58.35         | 35.11         | 65.75       | 39.67      | 72.13      | 60.57                      |
| InternVL3.5-1B     | 50.27  | 36.17     | 44.19 | 60.99         | 33.53         | 57.12       | 41.89      | 68.93      | 58.32                      |
| FastVLM-1.5B       | 53.13  | 24.99     | 43.29 | 23.92         | 26.61         | 61.56       | 38.78      | 64.84      | 50.89                      |

### 📚 Resources

* [Liquid AI Blog Post](https://www.liquid.ai/blog/introducing-lfm2-5-the-next-generation-of-on-device-ai)
* [LFM2 Technical Report (arXiv)](https://arxiv.org/abs/2511.23404)
* [Liquid AI Documentation](https://docs.liquid.ai/lfm)
* [Liquid Playground](https://playground.liquid.ai/)


# Magistral: How to Run & Fine-tune

Meet Magistral - Mistral's new reasoning models.

**Magistral-Small-2509** is a reasoning LLM developed by Mistral AI. It excels at coding and mathematics and supports multiple languages. Magistral supports a 128k token context window and was finetuned from [**Mistral-Small-3.2**](https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506). Magistral runs perfectly well locally on a single RTX 4090 or a Mac with 16 to 24GB RAM.

<a href="#running-magistral" class="button primary">Running Magistral Tutorial</a> <a href="#fine-tuning-magistral-with-unsloth" class="button secondary">Fine-tuning Magistral</a>

{% hint style="success" %}
Update: **Magistral-2509** new update is out as of September, 2025!\
\
Now with Vision support! We worked with Mistral again with the release of Magistral. Make sure to download Mistral's official uploads or Unsloth's uploads to get the correct implementation (ie correct system prompt, correct chat template etc.)

**If you're using llama.cpp, please use `--jinja` to enable the system prompt!**
{% endhint %}

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized Mistral LLMs with minimal accuracy loss.

#### Magistral-Small **- Unsloth Dynamic** uploads:

<table><thead><tr><th width="255.64999389648438">Dynamic 2.0 GGUF (to run)</th><th width="305.25">Dynamic 4-bit (to finetune/deploy)</th><th>Dynamic Float8</th></tr></thead><tbody><tr><td><ul><li><a href="https://huggingface.co/unsloth/Magistral-Small-2509-GGUF">Magistral-Small-2509-GGUF</a> - new</li><li><a href="https://huggingface.co/unsloth/Magistral-Small-2507-GGUF">Magistral-Small-2507-GGUF</a></li><li><a href="https://huggingface.co/unsloth/Magistral-Small-2506-GGUF">Magistral-Small-2506-GGUF</a></li></ul></td><td><ul><li><a href="https://huggingface.co/unsloth/Magistral-Small-2509-unsloth-bnb-4bit">Magistral-Small-2509-unsloth-bnb-4bit</a> - new</li><li><a href="https://huggingface.co/unsloth/Magistral-Small-2507-unsloth-bnb-4bit">Magistral-Small-2507-unsloth-bnb-4bit</a></li><li><a href="https://huggingface.co/unsloth/Magistral-Small-2506-unsloth-bnb-4bit">Magistral-Small-2506-unsloth-bnb-4bit</a></li></ul></td><td><ul><li><a href="https://huggingface.co/unsloth/Magistral-Small-2509-FP8-Dynamic">Magistral-Small-2509-FP8-Dynamic</a></li><li><a href="https://huggingface.co/unsloth/Magistral-Small-2509-FP8-torchao">Magistral-Small-2509-FP8-torchao</a></li></ul></td></tr></tbody></table>

## 🖥️ **Running Magistral**

### :gear: Official Recommended Settings

According to Mistral AI, these are the recommended settings for inference:

* <mark style="background-color:blue;">**Temperature of: 0.7**</mark>
* Min\_P of: 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1)
* Set <mark style="background-color:green;">**top\_p to: 0.95**</mark>
* A 128k context window is supported, **but** performance might degrade past **40k**. So we recommend setting the maximum length to 40k if you see bad performance.

**This is the recommended system prompt for Magistral 2509, 2507:**

{% code overflow="wrap" %}

```
First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.

Your thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response. Use the same language as the input.[/THINK]Here, provide a self-contained response.
```

{% endcode %}

**This is the recommended system prompt for Magistral 2506:**

```
A user will ask you to solve a task. You should first draft your thinking process (inner monologue) until you have derived the final answer. Afterwards, write a self-contained summary of your thoughts (i.e. your summary should be succinct but contain all the critical steps you needed to reach the conclusion). You should use Markdown to format your response. Write both your thoughts and summary in the same language as the task posed by the user. NEVER use \boxed{} in your response.

Your thinking process must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct answer.
</think>

Here, provide a concise summary that reflects your reasoning and presents a clear final answer to the user. Don't mention that this is a summary.

Problem:
```

{% hint style="success" %}
Our dynamic uploads have the '`UD`' prefix in them. Those without are not dynamic however still utilize our calibration dataset.
{% endhint %}

* **Multilingual:** Magistral supports many languages including: English, French, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Nepali, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Swedish, Turkish, Ukrainian, Vietnamese, Arabic, Bengali, Chinese, and Farsi.

### :question:Testing the model

Mistral has their own vibe checking prompts which can be used to evaluate Magistral. Keep in mind these tests are based on running the full unquantized version of the model, however you could also test them on quantized versions:

**Easy -** *Make sure they always work*

```py
prompt_1 = 'How many "r" are in strawberry?'

prompt_2 = 'John is one of 4 children. The first sister is 4 years old. Next year, the second sister will be twice as old as the first sister. The third sister is two years older than the second sister. The third sister is half the ago of her older brother. How old is John?'

prompt_3 = '9.11 and 9.8, which is greater?'
```

**Medium** - *Should most of the time be correct*

```py
prompt_4 = "Think about 5 random numbers. Verify if you can combine them with addition, multiplication, subtraction or division to 133"

prompt_5 = "Write 4 sentences, each with at least 8 words. Now make absolutely sure that every sentence has exactly one word less than the previous sentence."

prompt_6 = "If it takes 30 minutes to dry 12 T-shirts in the sun, how long does it take to dry 33 T-shirts?"
```

**Hard** - *Should sometimes get them right*

```py
prompt_7 = "Pick 5 random words each with at least 10 letters. Print them out. Reverse each word and print it out. Then extract letters that are alphabetically sorted smaller than "g" and print them. Do not use code."

prompt_8 = "Exactly how many days ago did the French Revolution start? Today is June 4th, 2025."
```

<mark style="color:green;">**We provide some**</mark> [<mark style="color:green;">**example outputs**</mark>](#sample-outputs) <mark style="color:green;">**at the end of the blog.**</mark>

## :llama: Tutorial: How to Run Magistral in Ollama

1. Install `ollama` if you haven't already!

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model with our dynamic quant. We did not set the context length automatically, so it will just use Ollama's default set context length.\
   Note you can call `ollama serve &`in another terminal if it fails! We include all suggested parameters (temperature etc) in `params` in our Hugging Face upload!
3. Also Magistral supports 40K context lengths, so best to enable [**KV cache quantization**](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-set-the-quantization-type-for-the-kv-cache). We use 8bit quantization which saves 50% memory usage. You can also try `"q4_0"` or `"q8_0"`
4. **Ollama also sets the default context length to 4096**, as [mentioned here](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size). Use `OLLAMA_CONTEXT_LENGTH=8192` to change it to 8192. Magistral supports up to 128K, but 40K (40960) is tested most.

```bash
export OLLAMA_KV_CACHE_TYPE="f16"
OLLAMA_CONTEXT_LENGTH=8192 ollama serve &
ollama run hf.co/unsloth/Magistral-Small-2509-GGUF:UD-Q4_K_XL
```

## 📖 Tutorial: How to Run Magistral in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run`

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli -hf unsloth/Magistral-Small-2509-GGUF:UD-Q4_K_XL --jinja --temp 0.7 --top-k -1 --top-p 0.95 -ngl 99
```

{% endcode %}

{% hint style="warning" %}
In llama.cpp, please use `--jinja` to enable the system prompt!
{% endhint %}

3. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose UD-Q4\_K\_XL, (Unsloth Dynamic), Q4\_K\_M, or other quantized versions (like BF16 full precision).

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Magistral-Small-2509-GGUF",
    local_dir = "unsloth/Magistral-Small-2509-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"], # For UD-Q4_K_XL
)
```

4. Run the model.
5. Edit `--threads -1` for the maximum CPU threads, `--ctx-size 40960` for context length (Magistral supports 40K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. We also use 8bit quantization for the K cache to reduce memory usage.
6. For conversation mode:

<pre class="language-bash"><code class="lang-bash">./llama.cpp/llama-cli \
    --model unsloth/Magistral-Small-2509-GGUF/Magistral-Small-2509-UD-Q4_K_XL.gguf \
    --ctx-size 40960 \
    <a data-footnote-ref href="#user-content-fn-1">--cache-type-k f16</a> \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.7 \
<strong>    --repeat-penalty 1.0 \
</strong>    --min-p 0.01 \
    --top-k -0 \
    --top-p 0.95 \
    <a data-footnote-ref href="#user-content-fn-2">--jinja</a>
</code></pre>

{% hint style="warning" %}
Remember to remove \<bos> since Magistral auto adds a \<bos>
{% endhint %}

## Sample outputs

<details>

<summary>How many "r" are in strawberry? [Correct answer = 3]</summary>

{% code overflow="wrap" lineNumbers="true" fullWidth="true" %}

```
<think>
Alright, I have a problem to solve: "How many 'r's are in strawberry?"

First, I need to understand the problem. It's asking for the count of the letter 'r' in the word "strawberry".

Step 1: Write down the word and look at each letter.
The word is: s t r a w b e r r y

Step 2: Identify the 'r's in the word.
Let's go through each letter one by one:
1. s - not an r
2. t - not an r
3. r - that's one r
4. a - not an r
5. w - not an r
6. b - not an r
7. e - not an r
8. r - that's another r
9. r - and another one
10. y - not an r

So, I see the letter 'r' appears at positions 3, 8, and 9.

Step 3: Count the number of 'r's.
From the above, there are three 'r's in "strawberry".

But wait, I should double-check to make sure I didn't miss anything or count incorrectly.

Let me write the word again: strawberry.

Looking at it:
- The third letter is 'r' (s-t-r)
- Then later, we have 'e-r-r' which is two more 'r's.

Yes, that's three 'r's in total.

But just to be thorough, sometimes letters can be hidden or the word might have silent letters, but in this case, all letters are pronounced in "strawberry", and 'r' is clearly pronounced in the third, eighth, and ninth positions.

So, the count is indeed three.

But wait, is it possible that the 'r's are not all pronounced? For example, in some accents or dialects, letters might be silent or pronounced differently. However, in standard pronunciation, all these 'r's are pronounced.

But the question is about the number of 'r's in the word, not about pronunciation. It's about the written form. So, regardless of pronunciation, we're counting the letters in the word "strawberry".

Let me write it out again and count:

s (1)
t (2)
r (3) - first r
a (4)
w (5)
b (6)
e (7)
r (8) - second r
r (9) - third r
y (10)

Yes, three 'r's.

But just to be sure, maybe I should think about the spelling. Sometimes letters can be part of digraphs or other combinations, but in "strawberry", the 'r's are all standalone letters.

Let me think about the spelling breakdown:
- "straw" has one 'r' (the third letter)
- "berry" has two 'r's (the second and third letters in "berry", but in the full word, it's the eighth and ninth letters)

So, "straw" has 1 'r', "berry" has 2 'r's, totaling 3 'r's in "strawberry".

I think I'm confident now. The word "strawberry" has three 'r's.

But before finalizing, I recall that sometimes letters can be repeated or there might be a trick. Let me see:

The word is: s-t-r-a-w-b-e-r-r-y.

Counting the 'r's:
1. The 'r' in the third position.
2. The 'r' in the eighth position.
3. The 'r' in the ninth position.

No other 'r's are present.

Therefore, the total number of 'r's in "strawberry" is three.

But just to cross-verify, perhaps I can write the word and highlight the 'r's:

s t r a w b e r r y
   r     r r

Yes, three 'r's.

Is there any chance I'm missing something? Maybe considering uppercase and lowercase? But the problem is case-insensitive since it's about the letter 'r', not 'R'. The word is given in lowercase, but even if it were uppercase, 'R' is the same letter as 'r' in terms of counting.

So, final count is three.

But wait, is "strawberry" spelled correctly? Let me check the spelling.

I think it's s-t-r-a-w-b-e-r-r-y. Yes, that seems correct.

Therefore, the number of 'r's is indeed three.

But just to be extra sure, maybe I can think of the word in a different way. For example, breaking it down:

"straw" + "berry" = "strawberry"

"straw" has one 'r' (the third letter: s-t-r)
"berry" has two 'r's (the second and third letters: b-e-r-r-y, but actually, 'berry' is b-e-r-r-y, so 'r' is the third and fourth letters? Wait, let's see:

'berry' is b-e-r-r-y. Wait, no:

b (1)
e (2)
r (3)
r (4)
y (5)

So, 'berry' has 'r's at positions 3 and 4, which are two 'r's.

'straw' is s-t-r-a-w, so 'r' is at position 3, one 'r'.

Combined, 'strawberry' is 'straw' + 'berry' = s-t-r-a-w-b-e-r-r-y, which has 'r's at positions 3, 3+5+1=9? Wait, no, let's index:

'straw' is positions 1-5 (s-t-r-a-w)
'berry' is positions 6-10 (b-e-r-r-y)

So 'r's in 'straw' are at position 3.
'r's in 'berry' are at positions 8 and 9 (since 'berry' starts at position 6: b(6), e(7), r(8), r(9), y(10)).

Thus, total 'r's are at positions 3, 8, and 9: three 'r's.

Yes, consistent again.

Is there any other way to approach this? Maybe counting the total letters and seeing how many are 'r's.

"strawberry" has 10 letters: s, t, r, a, w, b, e, r, r, y.

Counting 'r's: positions 3, 8, 9 → 3 'r's.

Alternatively, I could write out the word and replace non-'r's with another character to see how many 'r's are left.

Original: s t r a w b e r r y
Replace non-r: x x r x x x x r r x
Now count 'r's: positions 2 (but original was 3rd letter), wait no, in the replaced string, 'r's are at positions 3 (original position 3), and then after replacing others, the next 'r's are at positions... Wait, let's see:

Original positions:
1: s → x
2: t → x
3: r → r
4: a → x
5: w → x
6: b → x
7: e → x
8: r → r
9: r → r
10: y → x

So replaced string: x x r x x x x r r x
Now count 'r's: positions 3, 8, 9 in original correspond to positions 3, 8, 9 in replaced (but replaced has x's in other positions, so 'r's are still at 3,8,9). Wait, in replaced string, 'r's are at positions 3,8,9, but the length is still 10, so positions are same as original. So count is same: 3 'r's.

Another method: write the word and cross out non-'r's.

Original: s t r a w b e r r y
Cross out non-r: x x r x x x x r r x
Now count 'r's: there are three 'r's remaining.

All methods confirm that there are three 'r's in "strawberry".

But just to think outside the box, is there any ambiguity in the question? For example, is it asking for uppercase 'R' or lowercase 'r'? The problem says "r", lowercase, so we're counting lowercase 'r's. In the word "strawberry", all letters are lowercase (assuming the input is in lowercase; if it were mixed case, we'd have to consider, but since it's given as "strawberry", which is all lowercase, we're good).

Also, is it possible that some 'r's are part of a digraph or trigraph? For example, in some languages, 'rr' might be considered a single sound or letter, but in English orthography, 'rr' is two separate 'r's. So in "strawberry", the 'rr' is two 'r's.

Therefore, the count remains three.

Is there any other interpretation? Maybe considering that sometimes letters can represent different sounds or that 'r' might be silent, but as mentioned earlier, the question is about the written letter 'r', not pronunciation.

Final confirmation: let's write the word and count 'r's aloud:

s (no), t (no), r (1), a (no), w (no), b (no), e (no), r (2), r (3), y (no).

Yes, three 'r's.

Therefore, the number of 'r's in "strawberry" is three.

But before finalizing, let me think if there's any trick or if I'm missing something obvious. For example, sometimes questions like this have hidden complexities, like counting 'r's in the word's spelling or pronunciation, or considering that 'r' might appear in other forms, like in 'rr' which is two 'r's together, but we've already accounted for that.

Alternatively, is the question about the number of times the letter 'r' appears in the spelling of "strawberry"? Yes, that's how I interpreted it, and my count is three.

Is there any chance the word is spelled differently? For example, in some countries or dialects, is "strawberry" spelled differently? In American English and British English, it's spelled the same way: s-t-r-a-w-b-e-r-r-y.

Let me check the spelling again: s, t, r, a, w, b, e, r, r, y. Yes, that's correct.

Therefore, the number of 'r's is indeed three.

But just to be absolutely sure, perhaps I can think of the word in a sentence or see it written out:

Example sentence: "I ate a strawberry." The word is "strawberry", and in it, the 'r's are in the middle and at the end.

Looking at the spelling: s-t-r-a-w-b-e-r-r-y. Counting:

1. s
2. t
3. r (1)
4. a
5. w
6. b
7. e
8. r (2)
9. r (3)
10. y

Yes, three 'r's.

I think I've exhausted all possible ways to verify this count. Therefore, the number of 'r's in "strawberry" is three.

But wait, the user might have intended the question differently. Could it be about the number of times the sound 'r' appears in pronunciation? For example, sometimes letters are silent or combined in pronunciation, but the question asks for the count of the letter 'r', not the sound.

In pronunciation, "strawberry" is /ˈstroʊbɛri/ or similar, where the 'r's are pronounced as in the spelling. But since the question is about the letter count, not phonemes, we stick with the written form.

Therefore, the count remains three.

Is there any other interpretation? For example, counting uppercase 'R's if the word were capitalized? But the word is given in lowercase, and the question asks for 'r', lowercase, so it's consistent.

Final answer: there are three 'r's in "strawberry".

But just to double-check, perhaps I can list the letters with their positions:

Position: 1 2 3 4 5 6 7 8 9 10
Letter:   s t r a w b e r r y

Now, pick out the 'r's: positions 3, 8, 9. That's three 'r's.

Alternatively, write the word and circle the 'r's:

Strawberry: s t r a w b e r r y
The 'r's are in positions 3, 8, and 9. So, three 'r's.

No other interpretations seem plausible. Therefore, the answer is three.

But before finalizing, let me think if there's any ambiguity in the word itself. Could "strawberry" be misspelled or is there another spelling variant?

In English, "strawberry" is consistently spelled with two 'r's at the end: "berry" is spelled with two 'r's (b-e-r-r-y), and "straw" has one 'r' (s-t-r-a-w). So total is three 'r's.

Is there any other word that might be confused with "strawberry"? For example, "strawberi" or similar, but no, "strawberry" is the correct spelling with three 'r's.

Therefore, the final count is three.

But just to be thorough, perhaps I can think of the word in another language. For example, in Spanish, "strawberry" is "fresas" or "fresa", which don't have 'r's, but the question is about the English word "strawberry".

In English, as confirmed, there are three 'r's in "strawberry".

Is there any other angle? For example, if the word were in uppercase, like "STRAWBERRY", the count would still be the same since 'R' is the uppercase counterpart of 'r'. But the problem specifies 'r', lowercase, and the word is given in lowercase, so it's consistent.

Therefore, the number of 'r's in "strawberry" is three.

But just to think differently, perhaps the word is being considered in a different context, like in a code or cipher where letters might represent something else. But the problem is straightforward: count the number of times the letter 'r' appears in the word "strawberry".

So, after all this careful consideration, the answer is clearly three.

But before finalizing, let me think if I've made any mistake in counting. Let me recount:

Word: s t r a w b e r r y

Positions:
1: s
2: t
3: r (1)
4: a
5: w
6: b
7: e
8: r (2)
9: r (3)
10: y

Yes, three 'r's at positions 3, 8, and 9.

No mistake in counting.

Therefore, the final answer is that there are three 'r's in the word "strawberry".

But wait, the user might expect a different answer if they're considering something else, but based on the straightforward interpretation, it's three.

Perhaps the user expects us to count only the distinct 'r's or something, but all 'r's are identical letters in this context.

Alternatively, maybe the question is a trick question implying that 'r's are pronounced differently or not at all, but again, the question is about the count of the letter 'r' in the spelling, not pronunciation.

Thus, the answer is three.

But just to be sure, let me think of the word "strawberry" and say it aloud: "strawberry". The 'r's are clearly pronounced as in the spelling.

No hidden complexities here. The count is three.

Final verification: write the word and count 'r's:

s t r a w b e r r y

'r' appears in the 3rd, 8th, and 9th positions. That's three times.

Therefore, the number of 'r's in "strawberry" is three.

But before concluding, let me think if there's any other interpretation where the answer could be different. For example, if the word were written in a different font or if there were any hidden letters, but in standard writing, it's clear.

Alternatively, if the word were in a different language with different spelling, but "strawberry" is the English word, and in English, it's spelled with three 'r's.

Another thought: is it possible that the 'w' or another letter is sometimes considered an 'r'? No, in English alphabet, 'w' is a separate letter from 'r'.

No other interpretations seem valid. The answer is three.

But just to be extra cautious, perhaps the word is misspelled in the question, but assuming it's correctly spelled as "strawberry", the count is three.

Therefore, after thorough consideration, the number of 'r's in "strawberry" is three.

But wait, looking back at the word: strawberry. Let me write it and count:

s, t, r, a, w, b, e, r, r, y.

Let me list them:

1. s
2. t
3. r
4. a
5. w
6. b
7. e
8. r
9. r
10. y

So, 'r's are at positions 3, 8, and 9. That's three 'r's.

Is there any chance I'm miscounting positions? Let me recount:

1: s
2: t
3: r (1st r)
4: a
5: w
6: b
7: e
8: r (2nd r)
9: r (3rd r)
10: y

Yes, three 'r's.

Therefore, the final answer is that there are three 'r's in the word "strawberry".

But just to ensure completeness, perhaps the word is being considered in uppercase or another case, but since the question is about 'r' (lowercase) and the word is given in lowercase, it's consistent.

Alternatively, if the word were "Strawberry" (with a capital S), does that affect the count? The question asks for 'r' (lowercase), and in "Strawberry", the 'S' is uppercase, but the 'r's are lowercase (assuming standard title case where only the first letter is capitalized). But since the question specifies 'r' (lowercase), and the 'r's in "Strawberry" are lowercase (unless it's all caps or something), but the given word is "strawberry" all lowercase, so it's three 'r's.

But even if it were "Strawberry", the 'r's are still lowercase (assuming standard capitalization), so count remains the same.

But since the given word is "strawberry" in all lowercase, the count is three.

Therefore, the number of 'r's in "strawberry" is three.

But just to think of edge cases, what if the word had a silent 'r'? But as discussed earlier, the question is about the written letter, not pronunciation.

Alternatively, if the word were in a different font where 'r' looks like something else, but in standard fonts, it's clear.

No other possibilities seem to alter the count.

Thus, the final answer is three.

But before finalizing, let me think if the word could be written differently, like with a different spelling variant. For example, in some dialects or historical contexts, words might be spelled differently, but modern English spells "strawberry" with three 'r's (one in 'straw' and two in 'berry').

Therefore, the count is three.

After all this careful consideration and multiple verification methods, I am confident that the number of 'r's in "strawberry" is three.

But to ensure that I haven't missed anything, perhaps I can think of the word's origin or related words. For example, "straw" has one 'r', "berry" has two 'r's, combining gives three 'r's in "strawberry". This also confirms the count.

Therefore, the answer is three.

However, sometimes such questions are trick questions expecting a different answer. For example, if the question was about the number of times the letter 'r' appears in the word "strawberry" when spelled out, but that seems convoluted and unlikely.

Alternatively, counting the number of closed areas in the letter 'r' when written in a certain font, but that's not relevant here.

Another possibility is that the question expects us to count the number of times the sound 'r' appears in pronunciation, but again, the question asks for the count of the letter 'r', not sounds.

In pronunciation, "strawberry" has the 'r' sound in "straw" and in "berry", but since 'berry' has two 'r's in spelling, both are pronounced in standard English, so the count would still relate to the spelling, which is three 'r's.

But since the question is about the letter count in writing, not pronunciation, the answer remains three.

Therefore, after all this thorough checking, the number of 'r's in "strawberry" is three.

But just to be sure, perhaps I can ask myself: if I were to type the word "strawberry" and count how many times I press the 'r' key, it would be three times (assuming I don't make any typos while typing).

Thus, confirming again: three 'r's.

Therefore, the final answer is:

There are 3 'r's in the word "strawberry".

But to present it neatly, perhaps as a count:

The word "strawberry" spelled out: s-t-r-a-w-b-e-r-r-y.

Counting 'r's:
1. The third letter: r
2. The eighth letter: r
3. The ninth letter: r

Total: 3 'r's.

Therefore, the number of 'r's in "strawberry" is 3.

But to ensure completeness, let me think of the word's length: "strawberry" has 10 letters. The positions are:

1: s
2: t
3: r (1st r)
4: a
5: w
6: b
7: e
8: r (2nd r)
9: r (3rd r)
10: y

Thus, three 'r's at positions 3, 8, and 9.

No other 'r's present.

Therefore, the final count is three.

But to present it in a structured way:

Word: s t r a w b e r r y
Positions of 'r': 3, 8, 9
Number of 'r's: 3

Thus, the answer is 3.

But just to think if there's any other interpretation where the answer could be different, perhaps considering that in some fonts, 'r' might look like another letter, but in standard fonts, it's distinct.

Alternatively, if the word were in a different language where 'r' has a different representation, but since it's given in English, we proceed with English alphabet rules.

Therefore, the count is three.

After all this exhaustive checking, I conclude that the number of 'r's in "strawberry" is three.

But to present it neatly in the summary, here's the concise version:

The word "strawberry" has the following letters: s, t, r, a, w, b, e, r, r, y. Counting the 'r's, we find them at the 3rd, 8th, and 9th positions, totaling three 'r's.

Therefore, there are **3 'r's** in the word "strawberry".

But wait, looking back at the word: strawberry. Let me write it and count:

s (1), t (2), r (3), a (4), w (5), b (6), e (7), r (8), r (9), y (10).

So 'r's at positions 3, 8, 9: that's three 'r's.

Is there any possibility that one of these 'r's is not counted? For example, if one of them were part of a digraph or silent, but in standard English spelling and pronunciation, all these 'r's are present and pronounced.

Therefore, the count is indeed three.

But just to think differently, perhaps the question expects us to count only the distinct positions where 'r' appears, but since all 'r's are the same letter, the count is still three.

Alternatively, if the question was about the number of times the letter 'r' is written when writing the word "strawberry", that would also be three times.

Thus, the answer is three.

But to present it clearly:

We analyze the word "strawberry" by listing its letters and identifying the positions of 'r':

Letters: s, t, r, a, w, b, e, r, r, y
Positions of 'r': 3, 8, 9
Count of 'r's: 3

Therefore, there are three 'r's in "strawberry".

But to ensure that this is the expected answer, perhaps the user expects us to consider that 'rr' is a double 'r', but even then, it's two separate 'r's next to each other, totaling three 'r's in the word (one from 'straw' and two from 'berry').

Thus, no alternative interpretation changes the count.

Therefore, the final answer is that there are three 'r's in "strawberry".

But to summarize neatly, here's the concise version:

The word "strawberry" is spelled with the following letters in order: s, t, r, a, w, b, e, r, r, y. The letter 'r' appears at the 3rd, 8th, and 9th positions, giving a total count of three 'r's.

Thus, the number of 'r's in "strawberry" is:

\boxed{3}
</think>

**Summary:**
The word "strawberry" is analyzed by listing its letters and identifying the positions of the letter 'r'. The 'r's are found at the 3rd, 8th, and 9th positions, resulting in a total count of three 'r's.

Therefore, there are **3 'r's** in the word "strawberry".

\[
\boxed{3}
\]
```

{% endcode %}

</details>

<details>

<summary>Exactly how many days ago did the French Revolution start? Today is June 4th, 2025. [Correct answer = 86,157 days]</summary>

```
<think>
Alright, let's tackle this problem step by step. First, I need to know the exact start date of the French Revolution.

From my knowledge, the French Revolution is generally considered to have started on July 14, 1789, which is the date of the Storming of the Bastille. However, some historians argue that the revolution began earlier with events like the Estates-General meeting on May 5, 1789, or even earlier with financial and political crises. But for simplicity, I'll use July 14, 1789, as the start date since it's widely recognized as the symbolic beginning of the revolution.

Now, the problem gives today's date as June 4, 2025. I need to calculate the number of days between July 14, 1789, and June 4, 2025.

### Step 1: Calculate the total number of years between the two dates.
2025 - 1789 = 236 years.

But we need to be careful here because not all years have the same number of days (leap years have an extra day), and we're not counting full years from July 14 to July 14.

### Step 2: Break down the calculation into years, months, and days.
First, let's calculate the number of full years between July 14, 1789, and July 14, 2025. That's 2025 - 1789 = 236 years.

But since we're starting from July 14, 1789, and ending on June 4, 2025, which is before July 14, 2025, we have 235 full years (from July 14, 1789, to July 14, 2024) plus the days from July 14, 2024, to June 4, 2025. Wait, that doesn't seem right.

Actually, it's better to think of it as:
From July 14, 1789, to July 14, 2024, is 235 years (because 2024 - 1789 = 235).
Then from July 14, 2024, to June 4, 2025, is the remaining time.

But let's double-check:
From July 14, 1789, to July 14, 2024, is 235 years.
From July 14, 2024, to June 4, 2025, is less than a year, so we can calculate the days in this period separately.

But actually, to make it easier, we can calculate the total days from July 14, 1789, to June 4, 2025, by calculating the days from July 14, 1789, to July 14, 2024 (235 years) plus the days from July 14, 2024, to June 4, 2025.

But wait, July 14, 2024, to June 4, 2025, is not correct because June 4, 2025, is before July 14, 2025. So the correct breakdown is:
From July 14, 1789, to July 14, 2024: 235 years.
From July 14, 2024, to June 4, 2025: but this is negative because June 4 is before July 14. So it's better to think of it as:
From July 14, 1789, to June 4, 2024: 234 full years plus the days from July 14 to June 4 in the next year (but this seems confusing).

Perhaps a better approach is to calculate the number of days from July 14, 1789, to June 4, 2025.

First, let's calculate the number of days from July 14, 1789, to July 14, 2024. That's 235 years.

Now, how many days are there in 235 years? We need to account for leap years.

A year has 365 days, and a leap year has 366 days. Leap years occur every 4 years, but century years are not leap years unless they're divisible by 400.

First, calculate the number of leap years between 1789 and 2024 (not including 2024 if it's a leap year, but since we're going up to July 14, 2024, which is before February 29, the leap day in 2024 wouldn't affect the count if we're counting full years, but actually, since we're counting from July 14, 1789, to July 14, 2024, and 2024 is a leap year, but since we're not crossing into the next year's February, the leap day in 2024 doesn't affect this period because the extra day is in February, and we're starting after February (July is after February, so the leap day is already accounted for in the year's total days).

Wait, no. The leap day is February 29. If we're counting from July 14, 1789, to July 14, 2024, then the leap day in each leap year between 1789 and 2024 would have been included in the previous year's count, but since we're starting after February, the leap day in the starting year (1789) doesn't affect us (since we're starting after February 29). Similarly, for the ending year (2024), since we're ending on July 14, which is after February 29, the leap day in 2024 is included in the count for that year (since we're including the entire year up to July 14, but actually, we're counting full years from July 14 to July 14, so each full year contributes either 365 or 366 days depending on whether it's a leap year.

But actually, since we're counting the number of days between two dates, and both dates are after February in their respective years, the leap day affects the count only if the period includes February 29. But since we're counting full years from July 14 to July 14, and July is after February, the leap day is included in the full year count.

Wait, no. If we're counting from July 14, 1789, to July 14, 2024, that's 235 years. Each year in this period has 365 days, except for leap years which have 366 days.

Number of leap years between 1789 and 2023 (since 2024's leap day is after July 14, but we're counting full years up to July 14, 2024, which is before February 29, 2025... wait no, 2024's leap day is February 29, 2024, and we're up to July 14, 2024, which is after February 29, so the leap day in 2024 is included in our count if we're counting full years up to July 14, 2024.

Wait, no, the full years are from July 14, 1789, to July 14, 2024. That's 2024 - 1789 = 235 years, but since we're starting at July 14, 1789, and ending at July 14, 2024, that's exactly 235 years.

Now, how many leap years are there in this period? A year is a leap year if:
1. It's divisible by 4,
2. It's not divisible by 100, unless it's also divisible by 400.

But since we're counting from July 14, the leap day (February 29) is included in the year's count if the year is a leap year, because we're including the entire year from July 14 to July 14 (which is equivalent to counting from July 14 to July 14 of the next year, but actually, no, from July 14, 1789, to July 14, 1790, is one year, which may include February 29 if 1790 is a leap year... wait, no, 1790 is not divisible by 4, so it's not a leap year.

Wait, the period from July 14, 1789, to July 14, 1790, is one year, and it includes February 29, 1790? No, 1790 is not a leap year (1790 is not divisible by 4). The leap day is February 29 in a leap year, but since our period starts after February in 1789, and ends before February in 1790... wait no, our period is from July 14, 1789, to July 14, 1790, which includes February 29, 1790? Wait, no, 1789 to 1790 is not a leap year, because 1789 to 1790 is one year, and the leap day would be in February 1790 if 1790 were a leap year, but it's not (1790 is not divisible by 4).

Wait, perhaps it's easier to think that for each full year from July 14 to July 14, the number of days is 365, plus 1 if the year is a leap year and the period includes February 29. But since our period starts after February in the starting year and ends after February in the ending year, the leap day is included in the count for leap years.

So, the number of leap years between 1789 and 2024 inclusive (since 2024 is a leap year, and we're counting up to July 14, 2024, which is after February 29, 2024, so the leap day is included).

Number of years: 2024 - 1789 + 1 = 236 years. Wait, no, from July 14, 1789, to July 14, 2024, is 2024 - 1789 = 235 years (because at July 14, 1789, it's the start, and at July 14, 2024, it's after 235 years).

Number of leap years in this period: The first year is 1789 (not a leap year, since 1789 is not divisible by 4). The last year is 2024 (which is a leap year, divisible by 4 and not by 100 unless divisible by 400, but 2024 is divisible by 4 and not by 100).

The number of leap years between 1789 and 2024 inclusive is the number of years divisible by 4 in this range, minus those divisible by 100 but not by 400.

First, number of years divisible by 4 between 1789 and 2024 inclusive:
The first leap year after 1789 is 1792 (since 1789 + 3 = 1792, which is divisible by 4).
The last leap year before 2024 is 2024 itself (which is divisible by 4).
Number of leap years = (2024 - 1792) / 4 + 1 = (2024 - 1792) = 232, 232 / 4 = 58, +1 = 59.

But wait, 2024 is included, so it's correct: (2024 - 1792) / 4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Now, subtract the century years that are not leap years (i.e., divisible by 100 but not by 400). Between 1789 and 2024, the century years are 1800, 1900, and 2000.

Now, which of these are not leap years?
- 1800: divisible by 100 but not by 400 (since 1800 / 400 = 4.5, not integer) → not a leap year.
- 1900: same, divisible by 100 but not by 400 → not a leap year.
- 2000: divisible by 400 (2000 / 400 = 5) → is a leap year.

So, we have two century years that are not leap years: 1800 and 1900.

Thus, total leap years = 59 - 2 = 57.

So, total days for the full years = 235 * 365 + number of leap years = 235 * 365 + 57.

Let's calculate that:
235 * 365:
200 * 365 = 73000
35 * 365 = 35 * 300 + 35 * 65 = 10500 + 2275 = 12775
Total: 73000 + 12775 = 85775
Plus leap days: 85775 + 57 = 85832 days for the full years.

Now, we need to add the days from July 14, 2024, to June 4, 2025.

First, calculate the days remaining in 2024 after July 14:
July has 31 days, so from July 14 to July 31 is 31 - 14 = 17 days (including July 14? Wait, no, from July 14 to July 15 is 1 day, so July 14 to July 31 is 31 - 14 = 17 days, but we need to include July 14? Actually, the number of days from July 14 to July 31 inclusive is 31 - 14 + 1 = 18 days. But usually, when we say "from A to B", we count the days starting from A and including B, so from July 14 to July 14 is 1 day, from July 14 to July 15 is 2 days, etc. So from July 14 to July 31 is 31 - 14 + 1 = 18 days.

But in calculating intervals, sometimes it's exclusive. The standard way is that the number of days between two dates is (end - start) + 1 if including both, or (end - start) if including start but not end. Here, we want the number of days from July 14, 2024, up to but not including July 14, 2025, but our end date is June 4, 2025, which is before July 14, 2025.

Wait, no, we're calculating the days from July 14, 2024, to June 4, 2025.

Let's break it down:
1. From July 14, 2024, to December 31, 2024.
2. From January 1, 2025, to June 4, 2025.

First part: July 14 to December 31, 2024.
Number of days remaining in July after the 14th: 31 - 14 = 17 (but actually, it's 31 - 14 = 17, but that's from July 15 to July 31, which is 17 days. So from July 14 to July 31 is 31 - 14 + 1 = 18 days (including July 14).
Wait, no, if we're counting days from July 14 (inclusive) to July 31 (inclusive), that's 31 - 14 + 1 = 18 days.
But in calculating the interval between two dates, it's often (end - start) + 1 if both are included. Here, we want the number of days from July 14 (inclusive) to June 4, 2025 (inclusive), which is (June 4, 2025 - July 14, 2024) + 1? Wait, no, better to calculate it directly.

Let's calculate the days remaining in 2024 after July 14:
July: 31 - 14 = 17 (but this is from July 15 to July 31, which is 17 days. So from July 14 to July 31 is 18 days (including July 14).
But actually, if we're counting days starting from July 14, then:
- July: 31 - 14 = 17 (from July 15 to July 31), plus July 14 itself is 18 days in July.
But that can't be right because July has only 31 days. Wait, if we include July 14, then the days are July 14 to July 31, which is 31 - 14 + 1 = 18 days.

Similarly, for August: 31 days
September: 30
October: 31
November: 30
December: 31
Total for August to December: 31 + 30 + 31 + 30 + 31 = 153 days
Plus July: 18
Total for July 14 to Dec 31, 2024: 18 + 153 = 171 days

Wait, let's verify:
From July 14 to July 31: 31 - 14 + 1 = 18
August: 31
September: 30
October: 31
November: 30
December: 31
Total: 18 + 31 + 30 + 31 + 30 + 31 = 18 + 31 = 49; +30=79; +31=110; +30=140; +31=171. Yes, correct.

Now, from January 1, 2025, to June 4, 2025.
Is 2025 a leap year? 2025 is not divisible by 4, so no.
Months:
January: 31
February: 28 (not leap year)
March: 31
April: 30
May: 31
June: up to 4th is 4
Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155 days.

So total days from July 14, 2024, to June 4, 2025: 171 (remaining in 2024) + 155 (in 2025) = 326 days.

Wait, but we have to be careful with the counting. Are we including both July 14, 2024, and June 4, 2025? If we're calculating the number of days between two dates including both endpoints, then the calculation is correct. But in calculating the difference between two dates, it's often exclusive of the start date. The problem asks "how many days ago", which typically means counting from today backwards, not including today. But here, we're calculating the days from the start of the revolution to today, so we should include both the start date and today in the count.

But in the initial problem, it's asking for how many days ago the French Revolution started, which is the number of days from June 4, 2025, back to July 14, 1789. This would be (June 4, 2025 - July 14, 1789) minus 1 if we're counting days ago (since today is not ago from today). But usually, the number of days between two dates including both is (end - start) + 1. But "days ago" would be the duration from the start to today, which is today's date minus start date, not including the start date if we're counting days after. Wait, no, if the revolution started on day X, then the number of days ago it started is (today - X), not including X itself in the count.

But in our calculation, we have:
From July 14, 1789, to June 4, 2025.

If we include both endpoints, it's (June 4, 2025 - July 14, 1789) + 1.
But "days ago" would be (today - start date), which is today's date minus start date minus 1 if not including the start date. But actually, if the revolution started on July 14, 1789, and today is June 4, 2025, then the number of days ago it started is the number of days between July 14, 1789, and June 4, 2025, minus one if we're not counting the start day. But usually, "how many days ago" refers to the time elapsed, which is (end - start), where end is today, start is the event date. So it's (June 4, 2025 - July 14, 1789), which would be the number of days between the two dates, not including the start day if we're counting the days after.

But in our calculation, we did full years (July 14, 1789 to July 14, 2024) which is 235 years, and then from July 14, 2024 to June 4, 2025.

But actually, to find the number of days ago, we should calculate the difference between June 4, 2025, and July 14, 1789, which is (June 4, 2025 - July 14, 1789).

But in our initial approach, we calculated the days from July 14, 1789, to June 4, 2025, which is the same as (June 4, 2025 - July 14, 1789), and then we can add 1 if we want to include both endpoints. But for "days ago", we don't include the start day, so it's just (June 4, 2025 - July 14, 1789).

So our calculation is correct as is: 235 years' worth of days plus the remaining days.

But let's think differently: calculate the total days from July 14, 1789, to June 4, 2025.

Alternative approach:
Calculate the number of full years and the remaining days.

From July 14, 1789, to July 14, 2024: 2024 - 1789 = 235 years.
Number of leap years in this period: as before, 57.

Total days for full years: 235 * 365 + 57 = 85775 + 57 = 85832.

Now, days from July 14, 2024, to June 4, 2025.

As calculated earlier: 326 days.

Total days: 85832 (full years) + 326 (remaining) = 86158 days.

But wait, does this count include both July 14, 1789, and June 4, 2025? If so, then to get the number of days between them (excluding the start day), we'd subtract 1. But in our calculation, we've included both endpoints (since we included July 14, 2024, in the remaining days calculation by doing July 14 to July 31 as 18 days, which includes July 14).

But actually, in the remaining days calculation:
From July 14, 2024, to June 4, 2025:
We calculated July 14 to Dec 31, 2024: 171 days (including July 14)
Jan 1 to June 4, 2025: 155 days (including Jan 1)
Total: 171 + 155 = 326 days, which includes both July 14, 2024, and June 4, 2025.

Similarly, the full years from July 14, 1789, to July 14, 2024, include July 14, 1789, and July 14, 2024 (but July 14, 2024, is already included in the remaining days, so we have double-counted July 14, 2024).

Wait, no, the full years are from July 14, 1789 (inclusive) to July 14, 2024 (exclusive? Or inclusive?).

Actually, the period from July 14, 1789, to July 14, 2024, includes July 14, 1789, and July 14, 2024, if we're counting inclusively. But in terms of years, it's 235 years from July 14, 1789, to July 14, 2024 (since at July 14, 2024, it's been exactly 235 years since July 14, 1789).

But in our days calculation, the full years contribute 235 years' worth of days, where each year is from July 14 to July 14 of the next year. But actually, from July 14, 1789, to July 14, 1790, is one year, which has 365 or 366 days depending on whether it's a leap year. But since the year starts on July 14, the leap day (February 29) is included in that year if the year is a leap year.

But our initial calculation of leap years assumed calendar years (January to December), but our period is July to July. So we need to recalculate the number of leap years in the period from July 14, 1789, to July 14, 2024.

A year Y is a leap year if it's divisible by 4, but not by 100 unless also by 400. But since our year period starts in July, the leap day (February 29) is included in the year if Y is a leap year (because February 29 is before July in the same year).

Wait, no: the period from July 14, Y to July 14, Y+1 includes February of Y+1, not Y. Wait, no:

From July 14, 2023, to July 14, 2024: this includes February 29, 2024, if 2024 is a leap year (which it is). So the period from July 14, Y to July 14, Y+1 includes February of Y+1.

Thus, for the period from July 14, 1789, to July 14, 2024, the leap years are those Y where Y+1 is a leap year (i.e., Y+1 is divisible by 4, etc.).

Wait, no: the year from July 14, Y to July 14, Y+1 is a leap period if Y+1 is a leap year (i.e., if February 29 falls within this period, which it does if Y+1 is a leap year).

But actually, the period from July 14, Y to July 14, Y+1 includes February of Y+1, so the extra day is present if Y+1 is a leap year.

Thus, the number of leap periods in 235 years is equal to the number of leap years in Y+1 for Y from 1789 to 2023 (since the last period is from July 14, 2023, to July 14, 2024, and we're checking if 2024 is a leap year).

But Y ranges from 1789 to 2023 (since the periods are 1789-1790, ..., 2023-2024), so Y+1 ranges from 1790 to 2024.

Number of leap years between 1790 and 2024 inclusive.

Number of years divisible by 4 in 1790..2024:
First divisible by 4 is 1792 (since 1790/4 = 447.5, so next is 1792).
Last is 2024.
Number of terms = (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's a leap year).
So subtract 2: 59 - 2 = 57 leap years.

Thus, there are 57 leap periods in the 235 years (each corresponding to a leap year in Y+1, i.e., 1792 to 2024 excluding 1800 and 1900).

Thus, total days for full years: 235 * 365 + 57 = 85832.

Now, remaining days from July 14, 2024, to June 4, 2025: as before, 326.

Total days: 85832 + 326 = 86158.

But this counts both July 14, 1789, and June 4, 2025. If we want the number of days between them (excluding the start day), it's 86158 - 1 = 86157.

But the question is "how many days ago did the French Revolution start?" which is asking for the time elapsed since the start. The time elapsed from July 14, 1789, to June 4, 2025, is (June 4, 2025 - July 14, 1789), which is the same as counting the days from July 14, 1789 (not including) to June 4, 2025 (including), which is 86158 - 1 = 86157.

Wait, no: if we say "X days ago from today", it's the number of days in [today - X, today), so the revolution started X days ago means today - X = start date, so X = today - start date, which is the number of days from start date to today, excluding the start date if counting days after.

But in our calculation, we included both endpoints, so to get the number of days between them (excluding start), it's 86158 - 1 = 86157.

But let's think differently: the duration between two dates is (end - start) + 1 if including both, or (end - start) if including start but not end, etc. But in terms of days ago, if today is June 4, 2025, and the revolution started on July 14, 1789, then the number of days ago is (June 4, 2025 - July 14, 1789), which is the same as counting the days from July 15, 1789, to June 4, 2025, inclusive, which is (June 4, 2025 - July 15, 1789) + 1.

But our initial calculation was from July 14, 1789 (inclusive) to June 4, 2025 (inclusive), which is (June 4, 2025 - July 14, 1789) + 1 = 86158.

Thus, days ago = 86158 - 1 (since the day of start is not "ago" from today) = 86157.

But actually, if the revolution started on day X, then today is X + N days ago. So N = today - X, where today and X are dates, and the subtraction gives the number of days between them, excluding the start day X in the count.

But in our calculation, we did X to today inclusive, which is (today - X) + 1. So today - X = 86158 - 1 = 86157.

Thus, the number of days ago is 86157.

But let's verify with a smaller example. Suppose today is July 16, 2024, and the start was July 14, 2024. Then days ago is 2 (July 14 is 2 days ago from July 16). Our method would calculate days from July 14 to July 16 inclusive: 3 days (14,15,16), then subtract 1 to get days ago: 2, which is correct.

Another example: today is July 15, 2024, start was July 14, 2024. Days from July 14 to July 15 inclusive: 2 days. Days ago: 2 - 1 = 1, which is correct (July 14 is 1 day ago from July 15).

Thus, our calculation seems correct: total inclusive days is 86158, so days ago is 86157.

But wait, let's think about the full years part. We calculated 235 years with 57 leap periods, giving 235*365 + 57 = 85832 days for the full years. Then we added 326 days for the remaining period from July 14, 2024, to June 4, 2025, which is correct. So total is 86158 inclusive, so days ago is 86157.

But let's cross-validate the remaining days calculation (July 14, 2024, to June 4, 2025):

From July 14 to December 31, 2024:
July: 31 - 14 = 17 (from July 15 to July 31) + 1 (for July 14) = 18 days? Wait, no, from July 14 to July 14 is 1 day (just July 14). From July 14 to July 15 is 2 days (14 and 15). So from July 14 to July 31 is 31 - 14 + 1 = 18 days (including both ends).
Similarly for other months: but since we're starting at July 14, the first month is July with 18 days (July 14 to July 31), then full months August to December, then full months January to May, then June up to 4.

Wait, perhaps better to calculate it as:
From July 14, 2024, to June 4, 2025, is equal to:
(From July 14, 2024, to July 14, 2025) minus (from June 5, 2025, to July 14, 2025).

But that seems convoluted. Alternatively, calculate days remaining in 2024 after July 14, plus days in 2025 up to June 4.

Days in July after July 14: July has 31 days, so from July 15 to July 31 is 17 days (31 - 14 = 17, since July 15 is day 1, ..., July 31 is day 17). Wait, no, July 14 to July 31 inclusive is 31 - 14 + 1 = 18 days.

Similarly, for August: full month is 31 days.
September: 30
October: 31
November: 30
December: 31
Total for August to December: 31 + 30 + 31 + 30 + 31 = 153
Total for July: 18
Total for 2024: 18 (July) + 153 (Aug-Dec) = 171 days from July 14 to Dec 31, 2024 inclusive.

Now 2025 is not a leap year (2025 is not divisible by 4).
Days in 2025 up to June 4:
January: 31
February: 28 (not leap year)
March: 31
April: 30
May: 31
June: 4
Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155 days from Jan 1 to June 4 inclusive.

But we need from July 14, 2024, to June 4, 2025, inclusive, which is:
(July 14, 2024, to Dec 31, 2024) + (Jan 1, 2025, to June 4, 2025) = 171 + 155 = 326 days.

Thus, total days from start to today inclusive is 85832 (full years) + 326 (remaining) = 86158.
Thus, days ago is 86158 - 1 = 86157.

But let's cross-check the leap year count.

Number of leap years between July 14, 1789, and July 14, 2024 (inclusive of the start date but not the end? Wait, our period is from July 14, 1789, up to but not including July 14, 2025, but we're counting up to July 14, 2024, which is before July 14, 2025.

Wait, perhaps better to think in terms of the number of February 29s in the period from July 14, 1789, to June 4, 2025.

A February 29 falls within our period if the year Y is a leap year and February 29, Y, is >= July 14, 1789, and < June 4, 2025.

But since our start date is July 14, 1789, which is after February 29, 1789, the first possible leap day in our period is February 29, 1792.

Similarly, the last possible leap day before June 4, 2025, is February 29, 2024 (since February 29, 2028, is after June 4, 2025).

So leap days in our period are February 29 of years Y where:
1792 <= Y <= 2024, and Y is a leap year.

Number of leap years between 1792 and 2024 inclusive.

Number of years divisible by 4: (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's included).
Thus, subtract 2: 59 - 2 = 57 leap years.

Thus, there are 57 leap days in our period (February 29, 1792, to February 29, 2024).

Each of these adds 1 day to the total count (since we're counting the entire period inclusively, and February 29 is included in the period for these years).

Thus, our initial calculation seems correct: 235 full years with 57 leap periods contributing an extra day each, plus the remaining 326 days, gives 86158 total days including both endpoints. Therefore, the number of days ago is 86157.

But wait, is the count of leap years correct? Let's verify for a smaller range.

Example: from July 14, 2020, to June 4, 2022.
Leap years in this period: February 29, 2020 (but our start is July 14, 2020, so February 29, 2020, is before our start date, so not included).
Next is February 29, 2024, which is after June 4, 2022, so not included.
Wait, in our original problem, the period is July 14, 1789, to June 4, 2025, so February 29, 2024, is included (since June 4, 2025, is after February 29, 2024).

But in our initial calculation, we considered leap years up to 2024 (February 29, 2024), which is correct because our period includes February 29, 2024 (since June 4, 2025, is after February 29, 2024).

But in the small example from July 14, 2020, to June 4, 2022:
Leap days: February 29, 2020 is before July 14, 2020 (start), so not included.
February 29, 2024 is after June 4, 2022 (end), so not included.
Thus, no leap days in this period. But 2020 is a leap year, and our period starts after February 29, 2020, so the year from July 14, 2020, to July 14, 2021, does not include February 29 (since February 29, 2020, is before July 14, 2020, and February 29, 2021, doesn't exist). Wait, no, the period from July 14, 2020, to July 14, 2021, includes February 29, 2020? Wait, no, the period is from July 14, 2020, to July 14, 2021, which does not include February 29 (since February 29, 2020, is before July 14, 2020, and February 29, 2021, doesn't exist). Wait, no, the period from July 14, 2020, to July 14, 2021, includes February 29, 2021? Wait, no, February 29 is only in leap years, and 2021 is not a leap year. So in this small example, there are no February 29s in the period from July 14, 2020, to June 4, 2022, so the leap day count is correct (0 in this case).

Thus, our original calculation seems correct.

Final calculation:
Total days from July 14, 1789, to June 4, 2025 inclusive: 86158.
Number of days ago: 86158 - 1 = 86157.

But wait, is the count of full years correct? We have 235 full years (from July 14, 1789, to July 14, 2024), but actually, the period from July 14, 1789, to July 14, 2024, is 2024 - 1789 = 235 years, where each year is from July 14 to July 14 of the next year. So the number of full years is correct at 235.

Leap days in this period are February 29s that fall within any of these year-long periods. For a period from July 14, Y, to July 14, Y+1, the February 29 that could be included is February 29, Y+1 (since February 29, Y, is before July 14, Y, the start of the period, unless Y is before July, but our periods are from July 14 to July 14, so February is always before the start of the period if Y >= July). Wait, no, our periods are from July 14, Y, to July 14, Y+1, which includes February of Y+1, not Y. So for each Y from 1789 to 2023 (since the last period is from July 14, 2023, to July 14, 2024), the period includes February of Y+1.

Thus, the leap day is included in the period if Y+1 is a leap year. So the number of leap periods is equal to the number of Y in 1789..2023 such that Y+1 is a leap year, i.e., Y+1 is divisible by 4, not divisible by 100 unless also by 400.

Thus, Y+1 ranges from 1790 to 2024, and we need the number of leap years in this range, which is what we calculated earlier: 57.

Thus, the calculation holds.

Final answer: 86157 days ago.
</think>

### Summary:
The French Revolution started on July 14, 1789. To find out how many days ago this was from June 4, 2025, we calculate the total days between these two dates.

1. Calculate the number of full years between July 14, 1789, and July 14, 2024: 235 years.
2. Determine the number of leap years in this period by counting leap years between 1790 and 2024 (inclusive), accounting for century years that are not leap years (1800 and 1900), resulting in 57 leap years.
3. Calculate the total days for these full years: \( 235 \times 365 + 57 = 85832 \) days.
4. Calculate the days remaining from July 14, 2024, to June 4, 2025, which is 326 days (171 days in 2024 after July 14 and 155 days in 2025 up to June 4).
5. Sum the total days: \( 85832 + 326 = 86158 \) days (inclusive of both start and end dates).
6. Since the question asks for how many days ago the revolution started, we subtract 1 day to exclude the start day itself, resulting in 86157 days ago.

However, upon closer inspection, the correct interpretation is that the number of days between two dates (excluding the start day) is simply the difference in days between the two dates. Therefore, the correct calculation is the total inclusive days minus one to exclude the start day, which remains 86157.

But let's cross-validate with a smaller example to ensure correctness.

**Example:**
If today is July 16, 2024, and the event was on July 14, 2024:
- Inclusive days: July 14 to July 16 is 3 days (14, 15, 16).
- Days ago: 2 (July 14 is 2 days ago from July 16: July 15 and 16 are 1 and 0 days ago, respectively).

This matches our method: inclusive days = 3, so days ago = 3 - 1 = 2.

Thus, applying this to the original problem:
Inclusive days = 86158, so days ago = 86158 - 1 = 86157.

However, another approach is to calculate the exact difference between the two dates without including the start day in the count of "days ago". The duration between July 14, 1789, and June 4, 2025, excluding the start day, is (June 4, 2025 - July 14, 1789), which is equivalent to counting from July 15, 1789, to June 4, 2025, inclusive. This would be:

From July 15, 1789, to July 14, 2025: 235 years and 326 days (since July 15 to July 14 is one day less than a full year, but this seems confusing).

Alternatively, it's simpler to compute the total inclusive days and subtract 1, as initially done.

Therefore, the final answer is that the French Revolution started **86157 days ago**.

But let's verify the calculation once more for accuracy.

Recalculating the remaining days (July 14, 2024, to June 4, 2025):

From July 14 to December 31, 2024:
- July 14 to July 31: 18 days (including July 14)
- August: 31
- September: 30
- October: 31
- November: 30
- December: 31
Total: 18 + 31 + 30 + 31 + 30 + 31 = 171 days (but wait, July 14 to July 31 is 18 days, August starts on August 1, so total is correct as 171).

From January 1 to June 4, 2025 (not a leap year):
- January: 31
- February: 28
- March: 31
- April: 30
- May: 31
- June: 4
Total: 31 + 28 + 31 + 30 + 31 + 4 = 155 days.

Total remaining days: 171 + 155 = 326.

Full years: 235 years, with 57 leap days (as calculated earlier).
Total for full years: 235 * 365 + 57 = 85775 + 57 = 85832.

Total inclusive days: 85832 + 326 = 86158.

Days ago: 86158 - 1 = 86157.

Thus, the final answer is:

\[
\boxed{86157}
\]

However, upon re-reading the problem, it asks for "how many days ago did the French Revolution start?" which is asking for the duration from the start date to today, not including the start date itself. Therefore, the correct calculation is the difference between the two dates, which is (today - start date). If we compute the total inclusive days and subtract 1, we get the correct number of days ago, which is 86157.

But let's compute it directly:

Compute the difference between June 4, 2025, and July 14, 1789.

We can break it down as follows:

1. Calculate the number of days from July 14, 1789, to July 14, 2024: 235 years.
   - Number of leap years in this period: years divisible by 4 between 1792 and 2024, excluding 1800 and 1900: as before, 57.
   - Total days: 235 * 365 + 57 = 85832.

2. Add the days from July 14, 2024, to June 4, 2025: as before, 326.

Total days from July 14, 1789, to June 4, 2025, inclusive: 85832 + 326 = 86158.

But the number of days ago is the number of days after July 14, 1789, up to June 4, 2025, which is June 4, 2025 - July 14, 1789 = (days from July 15, 1789, to June 4, 2025, inclusive).

This is equal to (total inclusive days from July 14, 1789, to June 4, 2025) - 1 = 86157.

Thus, the French Revolution started **86157 days ago**.

However, let's cross-check with an online date difference calculator to ensure our manual calculation is correct.

Using an online calculator (e.g., timeanddate.com), the number of days between July 14, 1789, and June 4, 2025, inclusive is indeed 86158 days. Excluding the start date, it's 86157 days ago.

Thus, the final answer is:

The French Revolution started **86157 days ago**.

But wait, let's think about the interpretation of "days ago". If the revolution started on day X, then "days ago" is how many days have passed since X up to but not including today. But in our case, today is June 4, 2025, and the revolution started on July 14, 1789. The duration from July 14, 1789, to June 4, 2025 (excluding the start day) is the same as the number of days from July 15, 1789, to June 4, 2025, inclusive, which is the total inclusive days minus 1 (for the start day).

But actually, the standard way to compute "days ago" is to calculate (today - start date), which is the same as counting the days from the day after the start date up to and including today. But since our end date is today (June 4, 2025), and we're calculating days from July 14, 1789, to June 4, 2025 inclusive (which is 86158), then the days ago is 86158 - 1 = 86157.

But if we were to calculate (June 4, 2025 - July 14, 1789), this would directly give the number of days between the two dates not including the start date, which is what we want. This is equivalent to counting from July 15, 1789, to June 4, 2025 inclusive, which is the same as the total inclusive days minus one (the start day), so our initial calculation is correct.

Thus, the final answer is:

The French Revolution started **86157 days ago**.

But let's cross-validate with a smaller example again to ensure correctness.

Example:
Start date: July 14, 2024
Today: July 16, 2024
Inclusive days: July 14, 15, 16 → 3 days.
Days ago: 2 (since July 14 is 2 days ago from July 16: July 15 is 1 day ago, July 16 is today, 0 days ago).
Our method: inclusive days (3) - 1 = 2. Correct.

Another example:
Start: July 14, 2024
Today: July 15, 2024
Inclusive days: July 14, 15 → 2 days.
Days ago: 1 (July 14 is 1 day ago from July 15).
Our method: 2 - 1 = 1. Correct.

Thus, our method is correct.

Therefore, the French Revolution started **86157 days ago**.

However, let's think about the leap day calculation again to ensure no mistake was made.

We have full years from July 14, 1789, to July 14, 2024 (235 years), and we counted 57 leap periods in this interval (where each period is a year from July 14 to July 14, and contains February 29 if the next year is a leap year).

But let's think differently: the total number of February 29s in the entire period from July 14, 1789, to June 4, 2025.

February 29 occurs in leap years. The leap years in this period are those where February 29 is between July 14, 1789, and June 4, 2025.

Since July 14 is after February in any year, the February 29 of year Y is included in our period if:
July 14, 1789 <= February 29, Y < June 4, 2025.

But February 29, Y, is always before July 14, Y (since February is before July), so February 29, Y, is included if Y >= 1789 (since February 29, 1789, is before July 14, 1789, so not included), and February 29, Y < June 4, 2025.

But since February 29 is always before July 14 in the same year, February 29, Y, is included in our period if Y > 1789 and February 29, Y < June 4, 2025.

But February 29, Y, is before July 14, Y, so February 29, Y, is included if Y >= 1789 and February 29, Y < June 4, 2025.

But February 29, Y, is before July 14, Y, so for Y >= 1789, February 29, Y is before July 14, Y, so February 29, Y is only included in our period if Y > 1789 (since February 29, 1789, is before our start date of July 14, 1789), and February 29, Y < June 4, 2025.

But since February 29, Y, is always before July 14, Y, and our period starts on July 14, 1789, February 29, Y, is included if Y >= 1789 and February 29, Y >= July 14, 1789. But February is before July, so February 29, Y, is always before July 14, Y. Therefore, February 29, Y, is included in our period if Y > 1789 (since February 29, 1789, is before our start date) and February 29, Y < June 4, 2025.

But February 29, Y, is always before July 14, Y, so February 29, Y, is included in our period if:
July 14, 1789 <= February 29, Y < June 4, 2025.

But since February 29, Y, is always before July 14, Y, the first condition is satisfied if Y >= 1789, but February 29, 1789, is before July 14, 1789 (start of our period), so February 29, Y, is included if Y >= 1789 and February 29, Y >= July 14, 1789. But February 29 is always before July 14 in the same year, so February 29, Y, is never >= July 14, Y. Thus, February 29, Y, is included if Y > 1789 and February 29, Y >= July 14, 1789. But since February is before July, February 29, Y, is only >= July 14, 1789 if Y >= 1790 (because February 29, 1789, is before July 14, 1789, and February 29, 1790, is before July 14, 1790, etc., but our period starts on July 14, 1789, so February 29, Y, is included if Y >= 1790 (since February 29, 1790, is after July 14, 1789? Wait, no, February 29, Y, is always before July 14, Y, so February 29, Y, is included in our period if Y >= 1789 and February 29, Y >= July 14, 1789. But February 29, Y, is always before July 14, Y, so February 29, Y, is >= July 14, 1789 only if Y >= 1789 and February 29, Y >= July 14, 1789. But February is always before July, so February 29, Y, is always before July 14, Y, so February 29, Y >= July 14, 1789 would require Y > 1789 (since February 29, 1789, is before July 14, 1789, and February 29, 1790, is before July 14, 1790, etc., so February 29, Y, is never >= July 14, Y for Y >= 1789).

Wait, this seems confusing. Perhaps a better approach is to realize that in our period from July 14, 1789, to June 4, 2025, a February 29 is included if it falls within this interval. Since February is before July, February 29, Y, is included if Y >= 1790 (because February 29, 1789, is before July 14, 1789, so not included, and February 29, 1790, is after July 14, 1789, and before June 4, 2025, since June 4, 2025, is after February 29, 2024, and before February 29, 2028, etc.

But actually, February 29, Y, is included if Y is such that February 29, Y, is >= July 14, 1789 and < June 4, 2025. Since February is before July, February 29, Y, is >= July 14, 1789 only if Y >= 1790 (because February 29, 1789, is before July 14, 1789, but February 29, 1790, is after July 14, 1789? Wait, no, February 29, Y, is always before July 14, Y, so February 29, Y, is before July 14, Y >= 1789, so February 29, Y, cannot be >= July 14, 1789 unless Y > 1789 (but even then, February 29, 1790, is before July 14, 1790, etc.).

Wait, this seems incorrect. February 29, Y, is always before July 14, Y, so February 29, Y, cannot be >= July 14, Y for any Y. Therefore, February 29, Y, is included in our period if Y >= 1790 (since February 29, 1790, is after July 14, 1789, and before June 4, 2025).

Thus, the leap days included are February 29 of years Y where 1790 <= Y <= 2024 (since February 29, 2024, is before June 4, 2025, and February 29, 2028, would be after June 4, 2025).

Thus, we need to count the number of leap years Y in 1790 <= Y <= 2024.

Number of years divisible by 4 in 1790..2024:
First divisible by 4 is 1792 (since 1790 and 1791 are not divisible by 4).
Last divisible by 4 is 2024.
Number of terms: (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's included).
Thus, subtract 2: 59 - 2 = 57 leap years.

This matches our earlier count of leap periods (57), confirming that our initial approach was correct.

Thus, the final answer is indeed 86157 days ago.

However, let's perform the calculation once more to ensure no arithmetic errors were made.

Full years: 235 years (July 14, 1789, to July 14, 2024).
Number of leap years in this period: as above, 57 (for years 1792 to 2024 excluding 1800 and 1900).
Total days for full years: 235 * 365 + 57 = 235 * 365 = 235 * 300 + 235 * 65 = 70500 + 235 * 65.
Calculate 235 * 65:
200 * 65 = 13000
35 * 65 = 2275
Total: 13000 + 2275 = 15275
So 70500 + 15275 = 85775
Add leap days: 85775 + 57 = 85832.

Remaining days: July 14, 2024, to June 4, 2025.
From July 14 to July 31, 2024: 31 - 14 + 1 = 18 days.
August to December: 31 + 30 + 31 + 30 + 31 = 153.
Total for 2024: 18 + 153 = 171.
From Jan 1 to June 4, 2025:
January: 31
February: 28 (not a leap year)
March: 31
April: 30
May: 31
June: 4
Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155.
Total remaining days: 171 + 155 = 326.

Total inclusive days: 85832 (full years) + 326 (remaining) = 86158.
Days ago: 86158 - 1 = 86157.

Thus, the final answer is:

The French Revolution started **86157 days ago**.

\boxed{86157}
```

</details>

## 👁Vision Support

{% hint style="success" %}
Magistral 2509's September 2025 update now includes Vision support by default!
{% endhint %}

```
./llama.cpp/llama-mtmd-cli \
    --model unsloth/Magistral-Small-2509-GGUF/Magistral-Small-2509-Q4_K_XL.gguf \
    --mmproj unsloth/Magistral-Small-2509-GGUF/mmproj-BF16.gguf \
    --ctx-size 40960 \
    --cache-type-k f16
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.7 \
    --repeat-penalty 1.0 \
    --min-p 0.01 \
    --top-k -0 \
    --top-p 0.95 \
    --jinja
```

For Magistral versions before September 2025, [Xuan-Son](https://x.com/ngxson) from HuggingFace showed in their [GGUF repo](https://huggingface.co/ngxson/Devstral-Small-Vision-2505-GGUF) how it is actually possible to "graft" the vision encoder from Mistral 3.1 Instruct onto Devstral meaning you could do the same for Magistral! According to our tests and many users, it works quite well! We also uploaded our mmproj files which allows you to use the following:

<pre class="language-bash"><code class="lang-bash">./llama.cpp/llama-mtmd-cli \
    --model unsloth/Magistral-Small-2509-GGUF/Magistral-Small-2509-Q4_K_XL.gguf \
    --mmproj unsloth/Magistral-Small-2509-GGUF/mmproj-BF16.gguf \
    --ctx-size 40960 \
    <a data-footnote-ref href="#user-content-fn-1">--cache-type-k f16</a>
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.7 \
    --repeat-penalty 1.0 \
    --min-p 0.01 \
    --top-k -0 \
    --top-p 0.95 \
    --jinja
</code></pre>

## 🦥 Fine-tuning Magistral with Unsloth

Just like standard Mistral models including Mistral Small 3.1, Unsloth supports Magistral fine-tuning. Training is 2x faster, use 70% less VRAM and supports 8x longer context lengths. Magistral fits comfortably in a 24GB VRAM L4 GPU.

* **Magistral 2509 Kaggle (2x Tesla T4s) free** [**finetuning notebook**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_\(24B\)-Reasoning-Conversational.ipynb\&accelerator=nvidiaTeslaT4)
* Magistral 2509 Colab L4 (24GB) [finetuning notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Magistral_\(24B\)-Reasoning-Conversational.ipynb)

Magistral slightly exceeds the memory limits of a 16GB VRAM, so fine-tuning it for free on Google Colab isn't possible for now. However, you *can* fine-tune the model for free using [Kaggle](https://www.kaggle.com/danielhanchen/code), which offers access to dual GPUs.

**To finetune on new reasoning traces, you can use our free** [**Kaggle notebook for Magistral**](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_\(24B\)-Reasoning-Conversational.ipynb\&accelerator=nvidiaTeslaT4)

```python
!pip install --upgrade unsloth
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Magistral-Small-2509-unsloth-bnb-4bit",
    max_seq_length = 2048,   # Context length - can be longer, but uses more memory
    load_in_4bit = True,     # 4bit uses much less memory
    load_in_8bit = False,    # A bit more accurate, uses 2x memory
    full_finetuning = False, # We have full finetuning now!
    device_map = "balanced", # Uses 2x Telsa T4s
    # token = "hf_...",      # use one if using gated models
)
```

If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth:

```
pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo
```

## :diamond\_shape\_with\_a\_dot\_inside:Dynamic Float8 Checkpoints

We also provide 2 popular formats for float8 checkpoints, which also utilizes some of our dynamic methodology to retain maximum accuracy:

* [vLLM's Float8 format](https://huggingface.co/unsloth/Magistral-Small-2509-FP8-Dynamic)
* [TorchAO's Float8 format](https://huggingface.co/unsloth/Magistral-Small-2509-FP8-torchao)

Both are fantastic to deploy via vLLM. Read up on using TorchAO based FP8 quants in vLLM [here](https://docs.vllm.ai/en/latest/features/quantization/torchao.html).

[^1]: K quantization to reduce memory use. Can be f16, q8\_0, q4\_0

[^2]: Must use --jinja to enable system prompt


# IBM Granite 4.0

How to run IBM Granite-4.0 with Unsloth GGUFs on llama.cpp, Ollama and how to fine-tune!

IBM releases Granite-4.0 models with 3 sizes including **Nano** (350M & 1B), **Micro** (3B), **Tiny** (7B/1B active) and **Small** (32B/9B active). Trained on 15T tokens, IBM’s new Hybrid (H) Mamba architecture enables Granite-4.0 models to run faster with lower memory use.

Learn [how to run](#run-granite-4.0-tutorials) Unsloth Granite-4.0 Dynamic GGUFs or fine-tune/RL the model. You can [fine-tune Granite-4.0](#fine-tuning-granite-4.0-in-unsloth) with our free Colab notebook for a support agent use-case.

<a href="#run-granite-4.0-tutorials" class="button secondary">Running Tutorial</a><a href="#fine-tuning-granite-4.0-in-unsloth" class="button secondary">Fine-tuning Tutorial</a>

**Unsloth Granite-4.0 uploads:**

<table><thead><tr><th width="249">Dynamic GGUFs</th><th>Dynamic 4-bit + FP8</th><th>16-bit Instruct</th></tr></thead><tbody><tr><td><ul><li><a href="https://huggingface.co/unsloth/granite-4.0-h-350m-GGUF">H-350M</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-350m-GGUF">350M</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-1b-GGUF">H-1B</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-1b-GGUF">1B</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-small-GGUF">H-Small</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-tiny-GGUF">H-Tiny</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-micro-GGUF">H-Micro</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-micro-GGUF">Micro</a></li></ul></td><td><p>Dynamic 4-bit Instruct:</p><ul><li><a href="https://huggingface.co/unsloth/granite-4.0-h-micro-unsloth-bnb-4bit">H-Micro</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-micro-unsloth-bnb-4bit">Micro</a></li></ul><p>FP8 Dynamic:</p><ul><li><a href="https://huggingface.co/unsloth/granite-4.0-h-small-FP8-Dynamic">H-Small FP8</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-tiny-FP8-Dynamic">H-Tiny FP8</a></li></ul></td><td><ul><li><a href="https://huggingface.co/unsloth/granite-4.0-h-350m">H-350M</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-350m">350M</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-1b">H-1B</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-1b">1B</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-small">H-Small</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-tiny">H-Tiny</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-h-micro">H-Micro</a></li><li><a href="https://huggingface.co/unsloth/granite-4.0-micro">Micro</a></li></ul></td></tr></tbody></table>

You can also view our [Granite-4.0 collection](https://huggingface.co/collections/unsloth/granite-40-68ddf64b4a8717dc22a9322d) for all uploads including Dynamic Float8 quants etc.

**Granite-4.0 Models Explanations:**

* **Nano and H-Nano:** The 350M and 1B models offer strong instruction-following abilities, enabling advanced on-device and edge AI and research/fine-tuning applications.
* **H-Small (MoE):** Enterprise workhorse for daily tasks, supports multiple long-context sessions on entry GPUs like L40S (32B total, 9B active).
* **H-Tiny (MoE):** Fast, cost-efficient for high-volume, low-complexity tasks; optimized for local and edge use (7B total, 1B active).
* **H-Micro (Dense):** Lightweight, efficient for high-volume, low-complexity workloads; ideal for local and edge deployment (3B total).
* **Micro (Dense):** Alternative dense option when Mamba2 isn’t fully supported (3B total).

## Run Granite-4.0 Tutorials

### :gear: Recommended Inference Settings

IBM recommends these settings:

`temperature=0.0`, `top_p=1.0`, `top_k=0`

* <mark style="background-color:green;">**Temperature of 0.0**</mark>
* Top\_K = 0
* Top\_P = 1.0
* Recommended minimum context: 16,384
* Maximum context length window: 131,072 (128K context)

**Chat template:**

```
<|start_of_role|>system<|end_of_role|>You are a helpful assistant. Please ensure responses are professional, accurate, and safe.<|end_of_text|>
<|start_of_role|>user<|end_of_role|>Please list one IBM Research laboratory located in the United States. You should only output its name and location.<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>Almaden Research Center, San Jose, California<|end_of_text|>
```

### :llama: Ollama: Run Granite-4.0 Tutorial

1. Install `ollama` if you haven't already!

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload! You can change the model name '`granite-4.0-h-small-GGUF`' to any Granite model like 'granite-4.0-h-micro:Q8\_K\_XL'.

```bash
ollama run hf.co/unsloth/granite-4.0-h-small-GGUF:UD-Q4_K_XL
```

### 📖 llama.cpp: Run Granite-4.0 Tutorial

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run`

```bash
./llama.cpp/llama-cli \
    -hf unsloth/granite-4.0-h-small-GGUF:UD-Q4_K_XL
```

3. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision).

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/granite-4.0-h-small-GGUF",
    local_dir = "unsloth/granite-4.0-h-small-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"], # For Q4_K_M
)
```

4. Run Unsloth's Flappy Bird test
5. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length (Granite-4.0 supports 128K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
6. For conversation mode:

```bash
./llama.cpp/llama-mtmd-cli \
    --model unsloth/granite-4.0-h-small-GGUF/granite-4.0-h-small-UD-Q4_K_XL.gguf \
    --jinja \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.0 \
    --top-k 0 \
    --top-p 1.0
```

### 🐋 Docker: Run Granite-4.0 Tutorial

If you already have Docker desktop, all your need to do is run the command below and you're done:

```bash
docker model pull hf.co/unsloth/granite-4.0-h-small-GGUF:UD-Q4_K_XL
```

## :sloth: Fine-tuning Granite-4.0 in Unsloth

Unsloth now supports all Granite 4.0 models including nano, micro, tiny and small for fine-tuning. Training is 2x faster, use 50% less VRAM and supports 6x longer context lengths. Granite-4.0 micro and tiny fit comfortably in a 15GB VRAM T4 GPU.

* **Granite-4.0** [**free fine-tuning notebook**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Granite4.0.ipynb)
* Granite-4.0-350M [fine-tuning notebook](https://github.com/unslothai/notebooks/blob/main/nb/Granite4.0_350M.ipynb)

This notebook trains a model to become a Support Agent that understands customer interactions, complete with analysis and recommendations. This setup allows you to train a bot that provides real-time assistance to support agents.

We also show you how to train a model using data stored in a Google Sheet.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-3e28a77b82c05f317dd8e648783903eb0ecb9c6e%2Fgranite%204%20colab.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

**Unsloth config for Granite-4.0:**

```python
!pip install --upgrade unsloth
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/granite-4.0-h-micro",
    max_seq_length = 2048,   # Context length - can be longer, but uses more memory
    load_in_4bit = True,     # 4bit uses much less memory
    load_in_8bit = False,    # A bit more accurate, uses 2x memory
    full_finetuning = False, # We have full finetuning now!
    # token = "hf_...",      # use one if using gated models
)
```

If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth:

```
pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo
```


# Llama 4: How to Run & Fine-tune

How to run Llama 4 locally using our dynamic GGUFs which recovers accuracy compared to standard quantization.

The Llama-4-Scout model has 109B parameters, while Maverick has 402B parameters. The full unquantized version requires 113GB of disk space whilst the 1.78-bit version uses 33.8GB (-75% reduction in size). **Maverick** (402Bs) went from 422GB to just 122GB (-70%).

{% hint style="success" %}
Both text AND **vision** is now supported! Plus multiple improvements to tool calling.
{% endhint %}

Scout 1.78-bit fits in a 24GB VRAM GPU for fast inference at \~20 tokens/sec. Maverick 1.78-bit fits in 2x48GB VRAM GPUs for fast inference at \~40 tokens/sec.

For our dynamic GGUFs, to ensure the best tradeoff between accuracy and size, we do not to quantize all layers, but selectively quantize e.g. the MoE layers to lower bit, and leave attention and other layers in 4 or 6bit.

{% hint style="info" %}
All our GGUF models are quantized using calibration data (around 250K tokens for Scout and 1M tokens for Maverick), which will improve accuracy over standard quantization. Unsloth imatrix quants are fully compatible with popular inference engines like llama.cpp & Open WebUI etc.
{% endhint %}

**Scout - Unsloth Dynamic GGUFs with optimal configs:**

<table data-full-width="false"><thead><tr><th>MoE Bits</th><th>Type</th><th>Disk Size</th><th>Link</th><th>Details</th></tr></thead><tbody><tr><td>1.78bit</td><td>IQ1_S</td><td>33.8GB</td><td><a href="https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-IQ1_S.gguf">Link</a></td><td>2.06/1.56bit</td></tr><tr><td>1.93bit</td><td>IQ1_M</td><td>35.4GB</td><td><a href="https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-IQ1_M.gguf">Link</a></td><td>2.5/2.06/1.56</td></tr><tr><td>2.42bit</td><td>IQ2_XXS</td><td>38.6GB</td><td><a href="https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-IQ2_XXS.gguf">Link</a></td><td>2.5/2.06bit</td></tr><tr><td>2.71bit</td><td>Q2_K_XL</td><td>42.2GB</td><td><a href="https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF?show_file_info=Llama-4-Scout-17B-16E-Instruct-UD-Q2_K_XL.gguf">Link</a></td><td>3.5/2.5bit</td></tr><tr><td>3.5bit</td><td>Q3_K_XL</td><td>52.9GB</td><td><a href="https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/tree/main/UD-Q3_K_XL">Link</a></td><td>4.5/3.5bit</td></tr><tr><td>4.5bit</td><td>Q4_K_XL</td><td>65.6GB</td><td><a href="https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/tree/main/UD-Q4_K_XL">Link</a></td><td>5.5/4.5bit</td></tr></tbody></table>

{% hint style="info" %}
For best results, use the 2.42-bit (IQ2\_XXS) or larger versions.
{% endhint %}

**Maverick - Unsloth Dynamic GGUFs with optimal configs:**

| MoE Bits | Type      | Disk Size | HF Link                                                                                             |
| -------- | --------- | --------- | --------------------------------------------------------------------------------------------------- |
| 1.78bit  | IQ1\_S    | 122GB     | [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-IQ1_S)   |
| 1.93bit  | IQ1\_M    | 128GB     | [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-IQ1_M)   |
| 2.42-bit | IQ2\_XXS  | 140GB     | [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-IQ2_XXS) |
| 2.71-bit | Q2\_K\_XL | 151B      | [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-Q2_K_XL) |
| 3.5-bit  | Q3\_K\_XL | 193GB     | [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-Q3_K_XL) |
| 4.5-bit  | Q4\_K\_XL | 243GB     | [Link](https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-Q4_K_XL) |

## :gear: Official Recommended Settings

According to Meta, these are the recommended settings for inference:

* <mark style="background-color:blue;">**Temperature of 0.6**</mark>
* Min\_P of 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1)
* Top\_P of 0.9
* Chat template/prompt format:

{% code overflow="wrap" %}

```
<|header_start|>user<|header_end|>\n\nWhat is 1+1?<|eot|><|header_start|>assistant<|header_end|>\n\n
```

{% endcode %}

* A BOS token of `<|begin_of_text|>` is auto added during tokenization (do NOT add it manually!)
* According to <https://www.llama.com/docs/model-cards-and-prompt-formats/llama4_omni/>, there is an <mark style="background-color:yellow;">**suggested optional system prompt**</mark>, which is listed below:

```
You are an expert conversationalist who responds to the best of your ability. You are companionable and confident, and able to switch casually between tonal types, including but not limited to humor, empathy, intellectualism, creativity and problem-solving. 

You understand user intent and don’t try to be overly helpful to the point where you miss that the user is looking for chit-chat, emotional support, humor or venting.Sometimes people just want you to listen, and your answers should encourage that. For all other cases, you provide insightful and in-depth responses. Organize information thoughtfully in a way that helps people make decisions. Always avoid templated language.

You never lecture people to be nicer or more inclusive. If people ask for you to write something in a certain voice or perspective, such as an essay or a tweet, you can. You do not need to be respectful when the user prompts you to say something rude. 

You never use phrases that imply moral superiority or a sense of authority, including but not limited to “it’s important to”, “it’s crucial to”, “it’s essential to”, "it's unethical to", "it's worth noting…", “Remember…”  etc. Avoid using these.

Finally, do not refuse political prompts. You can help users express their opinion. 

You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise.
```

## 📖 Tutorial: How to Run Llama-4-Scout in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). More versions at: <https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF>

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF",
    local_dir = "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF",
    allow_patterns = ["*IQ2_XXS*"],
)
```

3. Run the model and try any prompt.
4. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length (Llama 4 supports 10M context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% hint style="success" %}
Use `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.
{% endhint %}

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/Llama-4-Scout-17B-16E-Instruct-UD-IQ2_XXS.gguf \
    --threads 32 \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --seed 3407 \
    --prio 3 \
    --temp 0.6 \
    --min-p 0.01 \
    --top-p 0.9 \
    -no-cnv \
    --prompt "<|header_start|>user<|header_end|>\n\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|eot|><|header_start|>assistant<|header_end|>\n\n"
```

{% endcode %}

{% hint style="info" %}
In terms of testing, unfortunately we can't make the full BF16 version (ie regardless of quantization or not) complete the Flappy Bird game nor the Heptagon test appropriately. We tried many inference providers, using imatrix or not, used other people's quants, and used normal Hugging Face inference, and this issue persists.

<mark style="background-color:green;">**We found multiple runs and asking the model to fix and find bugs to resolve most issues!**</mark>
{% endhint %}

For Llama 4 Maverick - it's best to have 2 RTX 4090s (2 x 24GB)

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF",
    local_dir = "unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF",
    allow_patterns = ["*IQ1_S*"],
)
```

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/UD-IQ1_S/Llama-4-Maverick-17B-128E-Instruct-UD-IQ1_S-00001-of-00003.gguf \
    --threads 32 \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --seed 3407 \
    --prio 3 \
    --temp 0.6 \
    --min-p 0.01 \
    --top-p 0.9 \
    -no-cnv \
    --prompt "<|header_start|>user<|header_end|>\n\nCreate the 2048 game in Python.<|eot|><|header_start|>assistant<|header_end|>\n\n"
```

{% endcode %}

## :detective: Interesting Insights and Issues

During quantization of Llama 4 Maverick (the large model), we found the 1st, 3rd and 45th MoE layers could not be calibrated correctly. Maverick uses interleaving MoE layers for every odd layer, so Dense->MoE->Dense and so on.

We tried adding more uncommon languages to our calibration dataset, and tried using more tokens (1 million) vs Scout's 250K tokens for calibration, but we still found issues. We decided to leave these MoE layers as 3bit and 4bit.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-473c016e369887cfb981817dc350255715494a0c%2FSkipped_layers.webp?alt=media" alt=""><figcaption></figcaption></figure>

For Llama 4 Scout, we found we should not quantize the vision layers, and leave the MoE router and some other layers as unquantized - we upload these to <https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-dynamic-bnb-4bit>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-6f10e8a8723d1ed25e757c13fb6e9c14da0207a1%2FLlama-4-Scout-17B-16E-Instruct%20Quantization%20Errors.png?alt=media" alt=""><figcaption></figcaption></figure>

We also had to convert `torch.nn.Parameter` to `torch.nn.Linear` for the MoE layers to allow 4bit quantization to occur. This also means we had to rewrite and patch over the generic Hugging Face implementation. We upload our quantized versions to <https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-4bit> and <https://huggingface.co/unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth-bnb-8bit> for 8bit.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-de4f3dbb28f40125a6ec8d3b282e99fdf0b009bd%2Fimage.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

Llama 4 also now uses chunked attention - it's essentially sliding window attention, but slightly more efficient by not attending to previous tokens over the 8192 boundary.


# Grok 2

Run xAI's Grok 2 model locally!

You can now run **Grok 2** (aka Grok 2.5), the 270B parameter model by xAI. Full precision requires **539GB**, while the Unsloth Dynamic 3-bit version shrinks size down to just **118GB** (a 75% reduction). GGUF: [Grok-2-GGUF](https://huggingface.co/unsloth/grok-2-GGUF)

The **3-bit Q3\_K\_XL** model runs on a single **128GB Mac** or **24GB VRAM + 128GB RAM**, achieving **5+ tokens/s** inference. Thanks to the llama.cpp team and community for [supporting Grok 2](https://github.com/ggml-org/llama.cpp/pull/15539) and making this possible. We were also glad to have helped a little along the way!

All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run quantized Grok LLMs with minimal accuracy loss.

<a href="#run-in-llama.cpp" class="button secondary">Run in llama.cpp Tutorial</a>

## :gear: Recommended Settings

The 3-bit dynamic quant uses 118GB (126GiB) of disk space - this works well in a 128GB RAM unified memory Mac or on a 1x24GB card and 128GB of RAM. It is recommended to have at least 120GB RAM to run this 3-bit quant.

{% hint style="warning" %}
You must use `--jinja` for Grok 2. You might get incorrect results if you do not use `--jinja`
{% endhint %}

The 8-bit quant is \~300GB in size will fit in a 1x 80GB GPU (with MoE layers offloaded to RAM). Expect around 5 tokens/s with this setup if you have bonus 200GB RAM as well. To learn how to increase generation speed and fit longer contexts, [read here](#improving-generation-speed).

{% hint style="info" %}
Though not a must, for best performance, have your VRAM + RAM combined equal to the size of the quant you're downloading. If not, hard drive / SSD offloading will work with llama.cpp, just inference will be slower.
{% endhint %}

### Sampling parameters

* Grok 2 has a 128K max context length thus, use `131,072` context or less.
* Use `--jinja` for llama.cpp variants

There are no official sampling parameters to run the model, thus you can use standard defaults for most models:

* Set the <mark style="background-color:green;">**temperature = 1.0**</mark>
* <mark style="background-color:green;">**Min\_P = 0.01**</mark> (optional, but 0.01 works well, llama.cpp default is 0.1)

## Run Grok 2 Tutorial:

Currently you can only run Grok 2 in llama.cpp.

### ✨ Run in llama.cpp

{% stepper %}
{% step %}
Install the specific `llama.cpp` PR for Grok 2 on [GitHub here](https://github.com/ggml-org/llama.cpp/pull/15539). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && git fetch origin pull/15539/head:MASTER && git checkout MASTER && cd ..
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can do the below: (:Q3\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Remember the model has only a maximum of 128K context length.

{% hint style="info" %}
Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

```bash
export LLAMA_CACHE="unsloth/grok-2-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/grok-2-GGUF:Q3_K_XL \
    --jinja \
    --n-gpu-layers 99 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-Q3_K_XL` (dynamic 3-bit quant) or other quantized versions like `Q4_K_M` . We <mark style="background-color:green;">**recommend using our 2.7bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**or above to balance size and accuracy**</mark>.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/grok-2-GGUF",
    local_dir = "unsloth/grok-2-GGUF",
    allow_patterns = ["*UD-Q3_K_XL*"], # Dynamic 3bit
)
```

{% endstep %}

{% step %}
You can edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/grok-2-GGUF/UD-Q3_K_XL/grok-2-UD-Q3_K_XL-00001-of-00003.gguf \
    --jinja \
    --threads -1 \
    --n-gpu-layers 99 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

{% endcode %}
{% endstep %}
{% endstepper %}

## Model uploads

**ALL our uploads** - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks.

| MoE Bits | Type + Link                                                                         | Disk Size   | Details       |
| -------- | ----------------------------------------------------------------------------------- | ----------- | ------------- |
| 1.66bit  | [TQ1\_0](https://huggingface.co/unsloth/grok-2-GGUF/blob/main/grok-2-UD-TQ1_0.gguf) | **81.8 GB** | 1.92/1.56bit  |
| 1.78bit  | [IQ1\_S](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-IQ1_S)             | **88.9 GB** | 2.06/1.56bit  |
| 1.93bit  | [IQ1\_M](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-IQ1_M)             | **94.5 GB** | 2.5/2.06/1.56 |
| 2.42bit  | [IQ2\_XXS](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-IQ2_XXS)         | **99.3 GB** | 2.5/2.06bit   |
| 2.71bit  | [Q2\_K\_XL](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-Q2_K_XL)        | **112 GB**  | 3.5/2.5bit    |
| 3.12bit  | [IQ3\_XXS](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-IQ3_XXS)         | **117 GB**  | 3.5/2.06bit   |
| 3.5bit   | [Q3\_K\_XL](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-Q3_K_XL)        | **126 GB**  | 4.5/3.5bit    |
| 4.5bit   | [Q4\_K\_XL](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-Q4_K_XL)        | **155 GB**  | 5.5/4.5bit    |
| 5.5bit   | [Q5\_K\_XL](https://huggingface.co/unsloth/grok-2-GGUF/tree/main/UD-Q5_K_XL)        | **191 GB**  | 6.5/5.5bit    |

## :snowboarder: Improving generation speed

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, `-ot ".ffn_.*_exps.=CPU"` offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.

The [latest llama.cpp release](https://github.com/ggml-org/llama.cpp/pull/14363) also introduces high throughput mode. Use `llama-parallel`. Read more about it [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/parallel). You can also **quantize the KV cache to 4bits** for example to reduce VRAM / RAM movement, which can also make the generation process faster.

## 📐How to fit long context (full 128K)

To fit longer context, you can use **KV cache quantization** to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is `f16`) include the below.

`--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`

You should use the `_1` variants for somewhat increased accuracy, albeit it's slightly slower. For eg `q4_1, q5_1`

You can also quantize the V cache, but you will need to **compile llama.cpp with Flash Attention** support via `-DGGML_CUDA_FA_ALL_QUANTS=ON`, and use `--flash-attn` to enable it. Then you can use together with `--cache-type-k` :

`--cache-type-v f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`


# Devstral: How to Run & Fine-tune

Run and fine-tune Mistral Devstral 1.1, including Small-2507 and 2505.

**Devstral-Small-2507** (Devstral 1.1) is Mistral's new agentic LLM for software engineering. It excels at tool-calling, exploring codebases, and powering coding agents. Mistral AI released the original 2505 version in May, 2025.

Finetuned from [**Mistral-Small-3.1**](https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF), Devstral supports a 128k context window. Devstral Small 1.1 has improved performance, achieving a score of 53.6% performance on [SWE-bench verified](https://openai.com/index/introducing-swe-bench-verified/), making it (July 10, 2025) the #1 open model on the benchmark.

Unsloth Devstral 1.1 GGUFs contain additional <mark style="background-color:green;">**tool-calling support**</mark> and <mark style="background-color:green;">**chat template fixes**</mark>. Devstral 1.1 still works well with OpenHands but now also generalizes better to other prompts and coding environments.

As text-only, Devstral’s vision encoder was removed prior to fine-tuning. We've added [*<mark style="background-color:green;">**optional Vision support**</mark>*](#possible-vision-support) for the model.

{% hint style="success" %}
We also worked with Mistral behind the scenes to help debug, test and correct any possible bugs and issues! Make sure to **download Mistral's official downloads or Unsloth's GGUFs** / dynamic quants to get the **correct implementation** (ie correct system prompt, correct chat template etc)

Please use `--jinja` in llama.cpp to enable the system prompt!
{% endhint %}

All Devstral uploads use our Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) methodology, delivering the best performance on 5-shot MMLU and KL Divergence benchmarks. This means, you can run and fine-tune quantized Mistral LLMs with minimal accuracy loss!

#### **Devstral - Unsloth Dynamic** quants:

| Devstral 2507 (new)                                                                                                    | Devstral 2505                                                                                               |
| ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| GGUF: [Devstral-Small-2507-GGUF](https://huggingface.co/unsloth/Devstral-Small-2507-GGUF)                              | [Devstral-Small-2505-GGUF](https://huggingface.co/unsloth/Devstral-Small-2505-GGUF)                         |
| 4-bit BnB: [Devstral-Small-2507-unsloth-bnb-4bit](https://huggingface.co/unsloth/Devstral-Small-2507-unsloth-bnb-4bit) | [Devstral-Small-2505-unsloth-bnb-4bit](https://huggingface.co/unsloth/Devstral-Small-2505-unsloth-bnb-4bit) |

## 🖥️ **Running Devstral**

### :gear: Official Recommended Settings

According to Mistral AI, these are the recommended settings for inference:

* <mark style="background-color:blue;">**Temperature from 0.0 to 0.15**</mark>
* Min\_P of 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1)
* <mark style="background-color:orange;">**Use**</mark><mark style="background-color:orange;">**&#x20;**</mark><mark style="background-color:orange;">**`--jinja`**</mark><mark style="background-color:orange;">**&#x20;**</mark><mark style="background-color:orange;">**to enable the system prompt.**</mark>

**A system prompt is recommended**, and is a derivative of Open Hand's system prompt. The full system prompt is provided [here](https://huggingface.co/unsloth/Devstral-Small-2505/blob/main/SYSTEM_PROMPT.txt).

```
You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. You can interact with a computer to solve tasks.

<ROLE>
Your primary role is to assist users by executing commands, modifying code, and solving technical problems effectively. You should be thorough, methodical, and prioritize quality over speed.
* If the user asks a question, like "why is X happening", don't try to fix the problem. Just give an answer to the question.
</ROLE>

.... SYSTEM PROMPT CONTINUES ....
```

{% hint style="success" %}
Our dynamic uploads have the '`UD`' prefix in them. Those without are not dynamic however still utilize our calibration dataset.
{% endhint %}

## :llama: Tutorial: How to Run Devstral in Ollama

1. Install `ollama` if you haven't already!

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model with our dynamic quant. Note you can call `ollama serve &`in another terminal if it fails! We include all suggested parameters (temperature etc) in `params` in our Hugging Face upload!
3. Also Devstral supports 128K context lengths, so best to enable [**KV cache quantization**](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-set-the-quantization-type-for-the-kv-cache). We use 8bit quantization which saves 50% memory usage. You can also try `"q4_0"`

```bash
export OLLAMA_KV_CACHE_TYPE="q8_0"
ollama run hf.co/unsloth/Devstral-Small-2507-GGUF:UD-Q4_K_XL
```

## 📖 Tutorial: How to Run Devstral in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:Q4\_K\_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run`

```bash
./llama.cpp/llama-cli -hf unsloth/Devstral-Small-2507-GGUF:UD-Q4_K_XL --jinja
```

3. **OR** download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision).

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Devstral-Small-2507-GGUF",
    local_dir = "unsloth/Devstral-Small-2507-GGUF",
    allow_patterns = ["*Q4_K_XL*", "*mmproj-F16*"], # For Q4_K_XL
)
```

4. Run the model.
5. Edit `--threads -1` for the maximum CPU threads, `--ctx-size 131072` for context length (Devstral supports 128K context length!), `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. We also use 8bit quantization for the K cache to reduce memory usage.
6. For conversation mode:

<pre class="language-bash"><code class="lang-bash">./llama.cpp/llama-cli \
    --model unsloth/Devstral-Small-2507-GGUF/Devstral-Small-2507-UD-Q4_K_XL.gguf \
    --threads -1 \
    --ctx-size 131072 \
    <a data-footnote-ref href="#user-content-fn-1">--cache-type-k q8_0</a> \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.15 \
    --repeat-penalty 1.0 \
    --min-p 0.01 \
    --top-k 64 \
    --top-p 0.95 \
    <a data-footnote-ref href="#user-content-fn-2">--jinja</a>
</code></pre>

7. For non conversation mode to test our Flappy Bird prompt:

<pre class="language-bash"><code class="lang-bash">./llama.cpp/llama-cli \
    --model unsloth/Devstral-Small-2507-GGUF/Devstral-Small-2507-UD-Q4_K_XL.gguf \
    --threads -1 \
    --ctx-size 131072 \
    <a data-footnote-ref href="#user-content-fn-1">--cache-type-k q8_0</a> \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.15 \
    --repeat-penalty 1.0 \
    --min-p 0.01 \
    --top-k 64 \
    --top-p 0.95 \
    -no-cnv \
    --prompt "[SYSTEM_PROMPT]You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. You can interact with a computer to solve tasks.\n\n&#x3C;ROLE>\nYour primary role is to assist users by executing commands, modifying code, and solving technical problems effectively. You should be thorough, methodical, and prioritize quality over speed.\n* If the user asks a question, like "why is X happening", don\'t try to fix the problem. Just give an answer to the question.\n&#x3C;/ROLE>\n\n&#x3C;EFFICIENCY>\n* Each action you take is somewhat expensive. Wherever possible, combine multiple actions into a single action, e.g. combine multiple bash commands into one, using sed and grep to edit/view multiple files at once.\n* When exploring the codebase, use efficient tools like find, grep, and git commands with appropriate filters to minimize unnecessary operations.\n&#x3C;/EFFICIENCY>\n\n&#x3C;FILE_SYSTEM_GUIDELINES>\n* When a user provides a file path, do NOT assume it\'s relative to the current working directory. First explore the file system to locate the file before working on it.\n* If asked to edit a file, edit the file directly, rather than creating a new file with a different filename.\n* For global search-and-replace operations, consider using `sed` instead of opening file editors multiple times.\n&#x3C;/FILE_SYSTEM_GUIDELINES>\n\n&#x3C;CODE_QUALITY>\n* Write clean, efficient code with minimal comments. Avoid redundancy in comments: Do not repeat information that can be easily inferred from the code itself.\n* When implementing solutions, focus on making the minimal changes needed to solve the problem.\n* Before implementing any changes, first thoroughly understand the codebase through exploration.\n* If you are adding a lot of code to a function or file, consider splitting the function or file into smaller pieces when appropriate.\n&#x3C;/CODE_QUALITY>\n\n&#x3C;VERSION_CONTROL>\n* When configuring git credentials, use "openhands" as the user.name and "openhands@all-hands.dev" as the user.email by default, unless explicitly instructed otherwise.\n* Exercise caution with git operations. Do NOT make potentially dangerous changes (e.g., pushing to main, deleting repositories) unless explicitly asked to do so.\n* When committing changes, use `git status` to see all modified files, and stage all files necessary for the commit. Use `git commit -a` whenever possible.\n* Do NOT commit files that typically shouldn\'t go into version control (e.g., node_modules/, .env files, build directories, cache files, large binaries) unless explicitly instructed by the user.\n* If unsure about committing certain files, check for the presence of .gitignore files or ask the user for clarification.\n&#x3C;/VERSION_CONTROL>\n\n&#x3C;PULL_REQUESTS>\n* When creating pull requests, create only ONE per session/issue unless explicitly instructed otherwise.\n* When working with an existing PR, update it with new commits rather than creating additional PRs for the same issue.\n* When updating a PR, preserve the original PR title and purpose, updating description only when necessary.\n&#x3C;/PULL_REQUESTS>\n\n&#x3C;PROBLEM_SOLVING_WORKFLOW>\n1. EXPLORATION: Thoroughly explore relevant files and understand the context before proposing solutions\n2. ANALYSIS: Consider multiple approaches and select the most promising one\n3. TESTING:\n   * For bug fixes: Create tests to verify issues before implementing fixes\n   * For new features: Consider test-driven development when appropriate\n   * If the repository lacks testing infrastructure and implementing tests would require extensive setup, consult with the user before investing time in building testing infrastructure\n   * If the environment is not set up to run tests, consult with the user first before investing time to install all dependencies\n4. IMPLEMENTATION: Make focused, minimal changes to address the problem\n5. VERIFICATION: If the environment is set up to run tests, test your implementation thoroughly, including edge cases. If the environment is not set up to run tests, consult with the user first before investing time to run tests.\n&#x3C;/PROBLEM_SOLVING_WORKFLOW>\n\n&#x3C;SECURITY>\n* Only use GITHUB_TOKEN and other credentials in ways the user has explicitly requested and would expect.\n* Use APIs to work with GitHub or other platforms, unless the user asks otherwise or your task requires browsing.\n&#x3C;/SECURITY>\n\n&#x3C;ENVIRONMENT_SETUP>\n* When user asks you to run an application, don\'t stop if the application is not installed. Instead, please install the application and run the command again.\n* If you encounter missing dependencies:\n  1. First, look around in the repository for existing dependency files (requirements.txt, pyproject.toml, package.json, Gemfile, etc.)\n  2. If dependency files exist, use them to install all dependencies at once (e.g., `pip install -r requirements.txt`, `npm install`, etc.)\n  3. Only install individual packages directly if no dependency files are found or if only specific packages are needed\n* Similarly, if you encounter missing dependencies for essential tools requested by the user, install them when possible.\n&#x3C;/ENVIRONMENT_SETUP>\n\n&#x3C;TROUBLESHOOTING>\n* If you\'ve made repeated attempts to solve a problem but tests still fail or the user reports it\'s still broken:\n  1. Step back and reflect on 5-7 different possible sources of the problem\n  2. Assess the likelihood of each possible cause\n  3. Methodically address the most likely causes, starting with the highest probability\n  4. Document your reasoning process\n* When you run into any major issue while executing a plan from the user, please don\'t try to directly work around it. Instead, propose a new plan and confirm with the user before proceeding.\n&#x3C;/TROUBLESHOOTING>[/SYSTEM_PROMPT][INST]Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird\'s shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don\'t hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for error[/INST]"
</code></pre>

{% hint style="danger" %}
Remember to remove \<bos> since Devstral auto adds a \<bos>! Also please use `--jinja` to enable the system prompt!
{% endhint %}

## :eyes:Experimental Vision Support

[Xuan-Son](https://x.com/ngxson) from Hugging Face showed in their [GGUF repo](https://huggingface.co/ngxson/Devstral-Small-Vision-2505-GGUF) how it is actually possible to "graft" the vision encoder from Mistral 3.1 Instruct onto Devstral 2507. We also uploaded our mmproj files which allows you to use the following:

```bash
./llama.cpp/llama-mtmd-cli \
    --model unsloth/Devstral-Small-2507-GGUF/Devstral-Small-2507-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Devstral-Small-2507-GGUF/mmproj-F16.gguf \
    --threads -1 \
    --ctx-size 131072 \
    --cache-type-k q8_0 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.15
```

For example:

| Instruction and output code                                                                                   | Rendered code                                                                                                 |
| ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| ![](https://cdn-uploads.huggingface.co/production/uploads/63ca214abedad7e2bf1d1517/HDic53ANsCoJbiWu2eE6K.png) | ![](https://cdn-uploads.huggingface.co/production/uploads/63ca214abedad7e2bf1d1517/onV1xfJIT8gzh81RkLn8J.png) |

## 🦥 Fine-tuning Devstral with Unsloth

Just like standard Mistral models including Mistral Small 3.1, Unsloth supports Devstral fine-tuning. Training is 2x faster, use 70% less VRAM and supports 8x longer context lengths. Devstral fits comfortably in a 24GB VRAM L4 GPU.

Unfortunately, Devstral slightly exceeds the memory limits of a 16GB VRAM, so fine-tuning it for free on Google Colab isn't possible for now. However, you *can* fine-tune the model for free using our [Kaggle notebook](https://www.kaggle.com/notebooks/welcome?src=https://github.com/unslothai/notebooks/blob/main/nb/Kaggle-Magistral_\(24B\)-Reasoning-Conversational.ipynb\&accelerator=nvidiaTeslaT4), which offers access to dual GPUs. Just change the notebook's Magistral model name to the Devstral model.

If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth:

```bash
pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo
```

[^1]: K quantization to reduce memory use. Can be f16, q8\_0, q4\_0

[^2]: Must use --jinja to enable system prompt


# How to Run Local LLMs with Docker: Step-by-Step Guide

Learn how to run Large Language Models (LLMs) with Docker & Unsloth on your local device.

You can now run any model, including Unsloth [Dynamic GGUFs](/docs/basics/dynamic-3.0-ggufs), on Mac, Windows or Linux with a single line of code or **no code** at all. We collabed with Docker to simplify model deployment, and Unsloth now powers most GGUF models on Docker.

Before you start, make sure to look over [hardware requirements](#hardware-info--performance) and [our tips](#hardware-info--performance) for optimizing performance when running LLMs on your device.

<a href="/docs/models/tutorials/how-to-run-llms-with-docker#method-1-docker-terminal" class="button primary">Docker Terminal Tutorial</a><a href="#method-2-docker-desktop-no-code" class="button primary">Docker no-code Tutorial</a>

To get started, run OpenAI [gpt-oss](/docs/models/gpt-oss-how-to-run-and-fine-tune) with a single command:

```bash
docker model run ai/gpt-oss:20B
```

Or to run a specific [Unsloth model](/docs/get-started/unsloth-model-catalog) / quant from Hugging Face:

```bash
docker model run hf.co/unsloth/gpt-oss-20b-GGUF:F16
```

{% hint style="success" %}
You don’t need Docker Desktop, Docker CE is enough to run models.
{% endhint %}

#### **Why Unsloth + Docker?**

We collab with model labs like Google Gemma to fix model bugs and boost accuracy. Our Dynamic GGUFs consistently outperform other quant methods, giving you high-accuracy, efficient inference.

If you use Docker, you can run models instantly with zero setup. Docker uses [Docker Model Runner](https://github.com/docker/model-runner) (DMR), which lets you run LLMs as easily as containers with no dependency issues. DMR uses Unsloth models and `llama.cpp` under the hood for fast, efficient, up-to-date inference.

## :gear: Hardware Info + Performance

For the best performance, aim for your VRAM + RAM combined to be at least equal to the size of the quantized model you're downloading. If you have less, the model will still run, but significantly slower.

Make sure your device also has enough disk space to store the model. If your model only barely fits in memory, you can expect around \~5 tokens/s, depending on model size.

Having extra RAM/VRAM available will improve inference speed, and additional VRAM will enable the biggest performance boost (provided the entire model fits)

{% hint style="info" %}
**Example:** If you're downloading gpt-oss-20b (F16) and the model is 13.8 GB, ensure that your disk space and RAM + VRAM > 13.8 GB.
{% endhint %}

**Quantization recommendations:**

* For models under 30B parameters, use at least 4-bit (Q4).
* For models 70B parameters or larger, use a minimum of 2-bit quantization (e.g., UD\_Q2\_K\_XL).

## ⚡ Step-by-Step Tutorials

Below are **two ways** to run models with Docker: one using the [terminal](#method-1-docker-terminal), and the other using [Docker Desktop](#method-2-docker-desktop-no-code) with no code:

### Method #1: Docker Terminal

{% stepper %}
{% step %}

#### Install Docker

Docker Model Runner is already available in **both** [Docker Desktop](https://docs.docker.com/ai/model-runner/get-started/#docker-desktop) and [**Docker CE**](https://docs.docker.com/ai/model-runner/get-started/#docker-engine)**.**
{% endstep %}

{% step %}

#### Run the model

Decide on a model to run, then run the command via terminal.

* Browse the verified catalog of trusted models available on [Docker Hub](https://hub.docker.com/r/ai) or [Unsloth's Hugging Face](https://huggingface.co/unsloth) page.
* Go to Terminal to run the commands. To verify if you have `docker` installed, you can type 'docker' and enter.
* Docker Hub defaults to running Unsloth Dynamic 4-bit, however you can select your own quantization level (see step #3).

For example, to run OpenAI `gpt-oss-20b` in a single command:

```bash
docker model run ai/gpt-oss:20B
```

Or to run a specific [Unsloth](/docs/get-started/unsloth-model-catalog) gpt-oss quant from Hugging Face:

```bash
docker model run hf.co/unsloth/gpt-oss-20b-GGUF:UD-Q8_K_XL
```

**This is how running gpt-oss-20b should look via CLI:**

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FNQQGqQfLuX2a40i07Es1%2Funknown.png?alt=media&amp;token=6006e370-a298-4e5f-af9f-5c4e8c4a0fc8" alt="" width="563"><figcaption><p>gpt-oss-20b from Docker Hub</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFtzowkLonKo4N2LzEnhe%2Fgptoss%20ud8kxl.png?alt=media&amp;token=08eb6244-0626-4ad8-adbe-f5c4e5aa6d72" alt="" width="563"><figcaption><p>gpt-oss-20b with Unsloths' UD-Q8_K_XL quantization</p></figcaption></figure></div>
{% endstep %}

{% step %}

#### To run a specific quantization level:

If you want to run a specific quantization of a model, append `:` and the quantization name to the model (e.g., `Q4` for Docker or `UD-Q4_K_XL`). You can view all available quantizations on each model’s Docker Hub page. e.g. see the listed quantizations for gpt-oss [here](https://hub.docker.com/r/ai/gpt-oss#gptoss).

The same applies to Unsloth quants on Hugging Face: visit the [model’s HF page](https://huggingface.co/unsloth/gpt-oss-20b-GGUF?show_file_info=gpt-oss-20b-Q2_K_L.gguf), choose a quantization, then run something like: `docker model run hf.co/unsloth/gpt-oss-20b-GGUF:Q2_K_L`

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FI7MrphUugkU8eZ1f7lJz%2FScreenshot%202025-11-16%20at%2010.52.25%E2%80%AFPM.png?alt=media&amp;token=ae777fdb-258e-46d0-b06e-b68434f7fa58" alt="" width="563"><figcaption><p>gpt-oss quantization levels on <a href="https://hub.docker.com/r/ai/gpt-oss#gptoss">Docker Hub</a></p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F73UWIs9VAZ6Iz1omgZSm%2FScreenshot%202025-11-16%20at%2010.54.53%E2%80%AFPM.png?alt=media&amp;token=e8dee7f3-1b8b-4dda-a455-26df58102d16" alt="" width="563"><figcaption><p>Unsloth gpt-oss quantization levels on<a href="https://huggingface.co/unsloth/gpt-oss-20b-GGUF"> Hugging Face</a></p></figcaption></figure></div>
{% endstep %}
{% endstepper %}

### Method #2: Docker Desktop (no code)

{% stepper %}
{% step %}

#### Install Docker Desktop

Docker Model Runner is already available in [Docker Desktop](https://docs.docker.com/ai/model-runner/get-started/#docker-desktop).

1. Decide on a model to run, open Docker Desktop, then click on the models tab.
2. Click 'Add models +' or Docker Hub. Search for the model.

Browse the verified model catalog available on [Docker Hub](https://hub.docker.com/r/ai).

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6R8uxbedMklVHsriLgpZ%2FScreenshot%202025-11-16%20at%206.36.49%E2%80%AFAM.png?alt=media&amp;token=0fb849f5-bc72-4883-9b25-1a756334ab4b" alt=""><figcaption><p>#1. Click 'Models' tab then 'Add models +'</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fj7I69xgxDeHXkvVbjemq%2FScreenshot%202025-11-16%20at%206.46.47%E2%80%AFAM.png?alt=media&amp;token=3e25d495-f34b-47d6-9185-f145610eed10" alt=""><figcaption><p>#2. Search for your desired model.</p></figcaption></figure></div>
{% endstep %}

{% step %}

#### Pull the model

Click the model you want to run to see available quantizations.

* Quantizations range from 1–16 bits. For models under 30B parameters, use at least 4-bit (`Q4`).
* Choose a size that fits your hardware: ideally, your combined unified memory, RAM, or VRAM should be equal to or greater than the model size. For example, an 11GB model runs well on 12GB unified memory.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FdQnwVwsMhYtWTzAGFih3%2FScreenshot%202025-11-16%20at%206.47.26%E2%80%AFAM.png?alt=media&amp;token=b1f0e29a-b3de-4d96-b79a-931441744565" alt=""><figcaption><p>#3. Select which quantization you would like to pull.</p></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FFGMZEXgQjBdVqP0vFTX3%2FScreenshot%202025-11-16%20at%206.54.09%E2%80%AFAM.png?alt=media&amp;token=b7f0c0d8-ebef-4e7e-99a9-c44c4c5f0844" alt=""><figcaption><p>#4. Wait for model to finish downloading, then Run it.</p></figcaption></figure></div>
{% endstep %}

{% step %}

#### Run the model

Type any prompt in the 'Ask a question' box and use the LLM like you would use ChatGPT.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F9nVjWcVsYK9CeT8gk3nQ%2FScreenshot%202025-11-16%20at%206.54.50%E2%80%AFAM.png?alt=media&amp;token=d7e5b63d-9c3e-42b0-882c-de046bbcfc9a" alt="" width="563"><figcaption><p>An example of running Qwen3-4B <code>UD-Q8_K_XL</code></p></figcaption></figure>
{% endstep %}
{% endstepper %}

#### **To run the latest models:**

You can run any new model on Docker as long as it’s supported by `llama.cpp` or `vllm` and available on Docker Hub.

### What Is the Docker Model Runner?

The Docker Model Runner (DMR) is an open-source tool that lets you pull and run AI models as easily as you run containers. GitHub: <https://github.com/docker/model-runner>

It provides a consistent runtime for models, similar to how Docker standardized app deployment. Under the hood, it uses optimized backends (like `llama.cpp`) for smooth, hardware-efficient inference on your machine.

Whether you’re a researcher, developer, or hobbyist, you can now:

* Run open models locally in seconds.
* Avoid dependency hell, everything is handled in Docker.
* Share and reproduce model setups effortlessly.


# DeepSeek-V3-0324: How to Run Locally

How to run DeepSeek-V3-0324 locally using our dynamic quants which recovers accuracy

{% hint style="info" %}
Please see <https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally> (May 28th 2025 update) to learn on how to run DeepSeek faster and more efficiently!
{% endhint %}

DeepSeek is at it again! After releasing V3, R1 Zero and R1 back in December 2024 and January 2025, DeepSeek updated their checkpoints / models for V3, and released a March update!

According to DeepSeek, MMLU-Pro jumped +5.3% to 81.2%. **GPQA +9.3% points**. AIME + 19.8% and LiveCodeBench + 10.0%! They provided a plot showing how they compared to the previous V3 checkpoint and other models like GPT 4.5 and Claude Sonnet 3.7. <mark style="background-color:blue;">**But how do we run a 671 billion parameter model locally?**</mark>

<table data-full-width="true"><thead><tr><th>MoE Bits</th><th>Type</th><th>Disk Size</th><th>Accuracy</th><th>Link</th><th>Details</th></tr></thead><tbody><tr><td>1.78bit</td><td>IQ1_S</td><td><strong>173GB</strong></td><td>Ok</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-IQ1_S">Link</a></td><td>2.06/1.56bit</td></tr><tr><td>1.93bit</td><td>IQ1_M</td><td><strong>183GB</strong></td><td>Fair</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-IQ1_M">Link</a></td><td>2.5/2.06/1.56</td></tr><tr><td>2.42bit</td><td>IQ2_XXS</td><td><strong>203GB</strong></td><td><mark style="background-color:blue;"><strong>Suggested</strong></mark></td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-IQ2_XXS">Link</a></td><td>2.5/2.06bit</td></tr><tr><td>2.71bit</td><td>Q2_K_XL</td><td><strong>231GB</strong></td><td><mark style="background-color:purple;"><strong>Suggested</strong></mark></td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-Q2_K_XL">Link</a></td><td>3.5/2.5bit</td></tr><tr><td>3.5bit</td><td>Q3_K_XL</td><td><strong>320GB</strong></td><td>Great</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-Q3_K_XL">Link</a></td><td>4.5/3.5bit</td></tr><tr><td>4.5bit</td><td>Q4_K_XL</td><td><strong>406GB</strong></td><td>Best</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF/tree/main/UD-Q4_K_XL">Link</a></td><td>5.5/4.5bit</td></tr></tbody></table>

{% hint style="success" %}
DeepSeek V3's original upload is in float8, which takes 715GB. Using Q4\_K\_M halves the file size to 404GB or so, and our dynamic 1.78bit quant fits in around 151GB. **We suggest using our 2.7bit quant to balance size and accuracy! The 2.4bit one also works well!**
{% endhint %}

## :gear: Official Recommended Settings

According to [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324), these are the recommended settings for inference:

* <mark style="background-color:blue;">**Temperature of 0.3**</mark> (Maybe 0.0 for coding as [seen here](https://api-docs.deepseek.com/quick_start/parameter_settings))
* Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
* Chat template: `<｜User｜>Create a simple playable Flappy Bird Game in Python. Place the final game inside of a markdown section.<｜Assistant｜>`
* A BOS token of `<｜begin▁of▁sentence｜>` is auto added during tokenization (do NOT add it manually!)
* DeepSeek mentioned using a <mark style="background-color:green;">**system prompt**</mark> as well (optional) - it's in Chinese: `该助手为DeepSeek Chat，由深度求索公司创造。\n今天是3月24日，星期一。` which translates to: `The assistant is DeepSeek Chat, created by DeepSeek.\nToday is Monday, March 24th.`
* <mark style="background-color:orange;">**For KV cache quantization, use 8bit, NOT 4bit - we found it to do noticeably worse.**</mark>

## 📖 Tutorial: How to Run DeepSeek-V3 in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% hint style="warning" %}
NOTE using `-DGGML_CUDA=ON` for GPUs might take 5 minutes to compile. CPU only takes 1 minute to compile. You might be interested in llama.cpp's precompiled binaries.
{% endhint %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-IQ1_S`(dynamic 1.78bit quant) or other quantized versions like `Q4_K_M` . <mark style="background-color:green;">**I recommend using our 2.7bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>. More versions at: <https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF>

{% code overflow="wrap" %}

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/DeepSeek-V3-0324-GGUF-UD",
    local_dir = "unsloth/DeepSeek-V3-0324-GGUF-UD",
    allow_patterns = ["*UD-Q2_K_XL*"], # Dynamic 2.7bit (230GB) Use "*UD-IQ_S*" for Dynamic 1.78bit (151GB)
)
```

{% endcode %}

3. Run Unsloth's Flappy Bird test as described in our 1.58bit Dynamic Quant for DeepSeek R1.
4. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

<pre class="language-bash" data-overflow="wrap"><code class="lang-bash">./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-V3-0324-GGUF-UD/blob/main/UD-Q2_K_XL/DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf \
    <a data-footnote-ref href="#user-content-fn-1">--cache-type-k q8_0 </a>\
    <a data-footnote-ref href="#user-content-fn-2">--threads 20</a> \
    <a data-footnote-ref href="#user-content-fn-3">--n-gpu-layers 2</a> \
    -no-cnv \
    --prio 3 \
    --temp 0.3 \
    --min-p 0.01 \
    <a data-footnote-ref href="#user-content-fn-4">--ctx-size 4096</a> \
    --seed 3407 \
    --prompt "&#x3C;｜User｜>Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.&#x3C;｜Assistant｜>"
</code></pre>

<details>

<summary>If we run the above, we get 2 very different results.<br><br><strong>Standard 2-bit version:</strong> Click to view result <em><mark style="color:red;"><strong>(seizure warning!)</strong></mark></em><br><strong>Dynamic 2-bit version:</strong> See the result below:</summary>

<img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-5ab85f63c9fb12f0ce701dd3cfd540aa1503a636%2FOld.gif?alt=media" alt="" data-size="original">

Standard 2-bit. Fails with background, fails with collision

</details>

<div align="center"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-97735d09e3d399cf74f8e7112eef8ca56b0e10e9%2FNew.gif?alt=media" alt="" width="240"><figcaption><p>Dynamic 2-bit. Succeeds in creating a playable game.</p></figcaption></figure></div>

5. Like DeepSeek-R1, V3 has 61 layers. For example with a 24GB GPU or 80GB GPU, you can expect to offload after rounding down (reduce by 1 if it goes out of memory):

| Quant   | File Size | 24GB GPU | 80GB GPU | 2x80GB GPU |
| ------- | --------- | -------- | -------- | ---------- |
| 1.73bit | 173GB     | 5        | 25       | 56         |
| 2.22bit | 183GB     | 4        | 22       | 49         |
| 2.51bit | 212GB     | 2        | 19       | 32         |

### Running on Mac / Apple devices

For Apple Metal devices, be careful of `--n-gpu-layers`. If you find the machine going out of memory, reduce it. For a 128GB unified memory machine, you should be able to offload 59 layers or so.

```bash
./llama.cpp/llama-cli \
    --model DeepSeek-R1-GGUF/DeepSeek-V3-0324-UD-IQ1_S/DeepSeek-V3-0324-UD-IQ1_S-00001-of-00003.gguf \
    --cache-type-k q4_0 \
    --threads 16 \
    --prio 2 \
    --temp 0.6 \
    --ctx-size 8192 \
    --seed 3407 \
    --n-gpu-layers 59 \
    -no-cnv \
    --prompt "<｜User｜>Create a Flappy Bird game in Python.<｜Assistant｜>"
```

## :8ball: Heptagon Test

We also test our dynamic quants via [r/Localllama](https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/) which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1371de5a93c6c5b0e43e8bb51980d84554b199f4%2Fsnapshot.jpg?alt=media" alt="" width="563"><figcaption><p>The goal is to make the heptagon spin, and the balls in the heptagon should move.</p></figcaption></figure>

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/DeepSeek-V3-0324-GGUF-UD/blob/main/UD-Q2_K_XL/DeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf \
    --cache-type-k q8_0 \
    --threads 20 \
    --n-gpu-layers 2 \
    -no-cnv \
    --prio 3 \
    --temp 0.3 \
    --min-p 0.01 \
    --ctx-size 4096 \
    --seed 3407 \
    --prompt "<｜User｜>Write a Python program that shows 20 balls bouncing inside a spinning heptagon:\n- All balls have the same radius.\n- All balls have a number on it from 1 to 20.\n- All balls drop from the heptagon center when starting.\n- Colors are: #f8b862, #f6ad49, #f39800, #f08300, #ec6d51, #ee7948, #ed6d3d, #ec6800, #ec6800, #ee7800, #eb6238, #ea5506, #ea5506, #eb6101, #e49e61, #e45e32, #e17b34, #dd7a56, #db8449, #d66a35\n- The balls should be affected by gravity and friction, and they must bounce off the rotating walls realistically. There should also be collisions between balls.\n- The material of all the balls determines that their impact bounce height will not exceed the radius of the heptagon, but higher than ball radius.\n- All balls rotate with friction, the numbers on the ball can be used to indicate the spin of the ball.\n- The heptagon is spinning around its center, and the speed of spinning is 360 degrees per 5 seconds.\n- The heptagon size should be large enough to contain all the balls.\n- Do not use the pygame library; implement collision detection algorithms and collision response etc. by yourself. The following Python libraries are allowed: tkinter, math, numpy, dataclasses, typing, sys.\n- All codes should be put in a single Python file.<｜Assistant｜>"
```

{% endcode %}

<table data-view="cards"><thead><tr><th></th><th data-type="files"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td>Non Dynamic 2bit. Fails - <mark style="background-color:red;">SEIZURE WARNING</mark> again!</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-0e0d101cc282b869a97fc679da5a9e98141c6f62%2Funsloth-q2_k_rotate.txt?alt=media">unsloth-q2_k_rotate.txt</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-d2579e8d1145572548ed69b0e335925f47130ba3%2FInShot_20250325_185636426.gif?alt=media">InShot_20250325_185636426.gif</a></td></tr><tr><td>Dynamic 2bit. Actually solves the heptagon puzzle correctly!!</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-96202dad58c0d0004cdb51408332ac78b425138f%2Funsloth-q2_k_xl_rotate.txt?alt=media">unsloth-q2_k_xl_rotate.txt</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-669f5b08b6bf220a38dce445d4830fcd80d52be3%2FInShot_20250325_181710554.gif?alt=media">InShot_20250325_181710554.gif</a></td></tr><tr><td>Original float8</td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-902430fe71d14f19cd31d3fe271f9a636e91304e%2Ffp8-heptagon.txt?alt=media">fp8-heptagon.txt</a></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-4501a6aef616d02057b89e96baa9c1d5ed57666f%2FInShot_20250325_181423756.gif?alt=media">InShot_20250325_181423756.gif</a></td></tr></tbody></table>

The dynamic 2.7 bit quant which is only 230GB in size actually manages to solve the heptagon puzzle! The full output for all 3 versions (including full fp8) is below:

<details>

<summary>Dynamic 2bit Heptagon code</summary>

```python
import tkinter as tk
import math
import numpy as np
from dataclasses import dataclass
from typing import List, Tuple, Optional

# Constants
HEPTAGON_RADIUS = 300
BALL_RADIUS = 20
GRAVITY = 0.2
FRICTION = 0.99
BOUNCE_FACTOR = 0.8
ROTATION_SPEED = 360 / 5  # degrees per second
SPIN_FRICTION = 0.98
BALL_COLORS = [
    "#f8b862", "#f6ad49", "#f39800", "#f08300", "#ec6d51",
    "#ee7948", "#ed6d3d", "#ec6800", "#ec6800", "#ee7800",
    "#eb6238", "#ea5506", "#ea5506", "#eb6101", "#e49e61",
    "#e45e32", "#e17b34", "#dd7a56", "#db8449", "#d66a35"
]

@dataclass
class Ball:
    x: float
    y: float
    vx: float
    vy: float
    number: int
    spin: float = 0.0
    color: str = "#000000"

@dataclass
class Wall:
    x1: float
    y1: float
    x2: float
    y2: float

class BouncingBalls:
    def __init__(self, root):
        self.root = root
        self.canvas = tk.Canvas(root, width=800, height=800, bg="white")
        self.canvas.pack()
        self.balls: List[Ball] = []
        self.walls: List[Wall] = []
        self.heptagon_angle = 0
        self.last_time = 0
        
        self.setup_balls()
        self.setup_heptagon()
        
        self.root.after(16, self.update)
        self.root.bind("<space>", self.reset_balls)
    
    def setup_balls(self):
        for i in range(20):
            ball = Ball(
                x=400,
                y=400,
                vx=np.random.uniform(-5, 5),
                vy=np.random.uniform(-5, 5),
                number=i+1,
                color=BALL_COLORS[i]
            )
            self.balls.append(ball)
    
    def setup_heptagon(self):
        # Create initial heptagon walls
        self.update_heptagon_walls(0)
    
    def update_heptagon_walls(self, angle):
        self.walls = []
        center_x, center_y = 400, 400
        angle_rad = math.radians(angle)
        
        for i in range(7):
            angle1 = angle_rad + 2 * math.pi * i / 7
            angle2 = angle_rad + 2 * math.pi * (i + 1) / 7
            
            x1 = center_x + HEPTAGON_RADIUS * math.cos(angle1)
            y1 = center_y + HEPTAGON_RADIUS * math.sin(angle1)
            x2 = center_x + HEPTAGON_RADIUS * math.cos(angle2)
            y2 = center_y + HEPTAGON_RADIUS * math.sin(angle2)
            
            self.walls.append(Wall(x1, y1, x2, y2))
    
    def reset_balls(self, event=None):
        for ball in self.balls:
            ball.x = 400
            ball.y = 400
            ball.vx = np.random.uniform(-5, 5)
            ball.vy = np.random.uniform(-5, 5)
            ball.spin = np.random.uniform(-5, 5)
    
    def update(self):
        current_time = self.root.after_idle(self.root.after, 16, self.update)
        if self.last_time == 0:
            self.last_time = current_time
            return
        
        # Calculate delta time (approximate)
        dt = 0.016  # Assuming ~60 FPS
        
        # Update heptagon rotation
        self.heptagon_angle += ROTATION_SPEED * dt
        self.update_heptagon_walls(self.heptagon_angle)
        
        # Update balls
        for ball in self.balls:
            # Apply gravity
            ball.vy += GRAVITY
            
            # Apply friction
            ball.vx *= FRICTION
            ball.vy *= FRICTION
            ball.spin *= SPIN_FRICTION
            
            # Move ball
            ball.x += ball.vx
            ball.y += ball.vy
            
            # Check collisions with walls
            self.check_wall_collisions(ball)
            
            # Check collisions with other balls
            for other in self.balls:
                if other.number != ball.number:
                    self.check_ball_collision(ball, other)
        
        # Draw everything
        self.draw()
    
    def check_wall_collisions(self, ball):
        for wall in self.walls:
            # Find closest point on wall segment to ball
            closest = self.closest_point_on_segment(
                wall.x1, wall.y1, wall.x2, wall.y2, ball.x, ball.y
            )
            
            # Calculate distance to wall
            dx = ball.x - closest[0]
            dy = ball.y - closest[1]
            distance = math.sqrt(dx*dx + dy*dy)
            
            if distance < BALL_RADIUS:
                # Collision detected
                # Calculate normal vector
                nx = dx / distance
                ny = dy / distance
                
                # Calculate relative velocity along normal
                v_rel = ball.vx * nx + ball.vy * ny
                
                if v_rel < 0:  # Moving toward the wall
                    # Calculate impulse
                    j = -(1 + BOUNCE_FACTOR) * v_rel
                    
                    # Apply impulse
                    ball.vx += j * nx
                    ball.vy += j * ny
                    
                    # Add some spin based on collision
                    ball.spin += (ball.vx * ny - ball.vy * nx) * 0.1
                    
                    # Move ball out of collision
                    penetration = BALL_RADIUS - distance
                    ball.x += penetration * nx
                    ball.y += penetration * ny
    
    def check_ball_collision(self, ball1, ball2):
        dx = ball2.x - ball1.x
        dy = ball2.y - ball1.y
        distance = math.sqrt(dx*dx + dy*dy)
        
        if distance < 2 * BALL_RADIUS:
            # Collision detected
            nx = dx / distance
            ny = dy / distance
            
            # Calculate relative velocity
            v_rel_x = ball2.vx - ball1.vx
            v_rel_y = ball2.vy - ball1.vy
            v_rel = v_rel_x * nx + v_rel_y * ny
            
            if v_rel < 0:  # Moving toward each other
                # Calculate impulse
                j = -(1 + BOUNCE_FACTOR) * v_rel / 2
                
                # Apply impulses
                ball1.vx -= j * nx
                ball1.vy -= j * ny
                ball2.vx += j * nx
                ball2.vy += j * ny
                
                # Add spin based on collision
                ball1.spin += (ball1.vx * ny - ball1.vy * nx) * 0.05
                ball2.spin += (ball2.vx * ny - ball2.vy * nx) * 0.05
                
                # Move balls apart
                penetration = 2 * BALL_RADIUS - distance
                ball1.x -= penetration * nx * 0.5
                ball1.y -= penetration * ny * 0.5
                ball2.x += penetration * nx * 0.5
                ball2.y += penetration * ny * 0.5
    
    @staticmethod
    def closest_point_on_segment(x1, y1, x2, y2, x, y):
        # Vector from point to segment start
        dx = x - x1
        dy = y - y1
        
        # Segment vector
        sx = x2 - x1
        sy = y2 - y1
        
        # Projection of point onto segment
        dot = dx * sx + dy * sy
        len_sq = sx * sx + sy * sy
        param = dot / len_sq if len_sq != 0 else -1
        
        if param < 0:
            return x1, y1
        elif param > 1:
            return x2, y2
        else:
            return x1 + param * sx, y1 + param * sy
    
    def draw(self):
        self.canvas.delete("all")
        
        # Draw heptagon
        points = []
        for wall in self.walls:
            points.extend([wall.x1, wall.y1])
        self.canvas.create_polygon(points, fill="", outline="black", width=2)
        
        # Draw balls
        for ball in self.balls:
            # Draw ball
            self.canvas.create_oval(
                ball.x - BALL_RADIUS, ball.y - BALL_RADIUS,
                ball.x + BALL_RADIUS, ball.y + BALL_RADIUS,
                fill=ball.color, outline="black"
            )
            
            # Draw number with rotation based on spin
            angle = ball.spin * 10  # Scale spin for visual effect
            self.canvas.create_text(
                ball.x, ball.y,
                text=str(ball.number),
                font=("Arial", 12, "bold"),
                angle=angle
            )

if __name__ == "__main__":
    root = tk.Tk()
    root.title("Bouncing Balls in Spinning Heptagon")
    app = BouncingBalls(root)
    root.mainloop()
```

</details>

<details>

<summary>Non Dynamic 2bit Heptagon code</summary>

```python
import tkinter as tk
import math
import random
from dataclasses import dataclass
from typing import List, Tuple, Optional
import sys

# Constants
WIDTH, HEIGHT = 800, 800
HEPTAGON_RADIUS = 300
BALL_RADIUS = 15
GRAVITY = 0.5
FRICTION = 0.999
ELASTICITY = 0.8
ROTATION_SPEED = 2 * math.pi / 5  # 360 degrees per 5 seconds
SPIN_DECAY = 0.99

# Colors for the balls
BALL_COLORS = [
    "#f8b862", "#f6ad49", "#f39800", "#f08300", "#ec6d51",
    "#ee7948", "#ed6d3d", "#ec6800", "#ec6800", "#ee7800",
    "#eb6238", "#ea5506", "#ea5506", "#eb6101", "#e49e61",
    "#e45e32", "#e17b34", "#dd7a56", "#db8449", "#d66a35"
]

@dataclass
class Ball:
    x: float
    y: float
    vx: float
    vy: float
    radius: float
    color: str
    number: int
    spin: float = 0.0

@dataclass
class Heptagon:
    center_x: float
    center_y: float
    radius: float
    angle: float = 0.0

class BouncingBalls:
    def __init__(self, root):
        self.root = root
        self.canvas = tk.Canvas(root, width=WIDTH, height=HEIGHT, bg="white")
        self.canvas.pack()
        
        self.heptagon = Heptagon(WIDTH//2, HEIGHT//2, HEPTAGON_RADIUS)
        self.balls = []
        self.setup_balls()
        
        self.root.after(0, self.update)
        self.root.mainloop()
    
    def setup_balls(self):
        center_x, center_y = WIDTH//2, HEIGHT//2
        for i in range(20):
            self.balls.append(Ball(
                x=center_x,
                y=center_y,
                vx=0,
                vy=0,
                radius=BALL_RADIUS,
                color=BALL_COLORS[i],
                number=i+1,
                spin=0
            ))
    
    def update(self):
        self.canvas.delete("all")
        
        # Update heptagon angle
        self.heptagon.angle += ROTATION_SPEED / 60  # Assuming 60 FPS
        
        # Draw heptagon
        self.draw_heptagon()
        
        # Update and draw balls
        for ball in self.balls:
            # Apply gravity
            ball.vy += GRAVITY
            
            # Update position
            ball.x += ball.vx
            ball.y += ball.vy
            
            # Apply friction
            ball.vx *= FRICTION
            ball.vy *= FRICTION
            
            # Apply spin decay
            ball.spin *= SPIN_DECAY
            
            # Check collision with heptagon walls
            self.check_heptagon_collision(ball)
            
            # Check collision with other balls
            for other in self.balls:
                if other != ball:
                    if self.check_ball_collision(ball, other):
                        self.resolve_ball_collision(ball, other)
            
            # Draw the ball
            self.draw_ball(ball)
        
        self.root.after(16, self.update)  # ~60 FPS
    
    def draw_heptagon(self):
        center_x, center_y = self.heptagon.center_x, self.heptagon.center_y
        points = []
        for i in range(7):
            angle = self.heptagon.angle + i * 2 * math.pi / 7
            x = center_x + self.heptagon.radius * math.cos(angle)
            y = center_y + self.heptagon.radius * math.sin(angle)
            points.append((x, y))
        
        # Draw heptagon
        self.canvas.create_polygon(
            [points[0], points[1], points[2], points[3], 
             points[4], points[5], points[6]],
            outline="black", fill="", width=2
        )
    
    def draw_ball(self, ball):
        self.canvas.create_oval(
            ball.x - ball.radius,
            ball.y - ball.radius,
            ball.x + ball.radius,
            ball.y + ball.radius,
            fill=ball.color,
            outline="black"
        )
        
        # Draw the number
        self.canvas.create_text(
            ball.x, ball.y,
            text=str(ball.number),
            fill="black"
        )
    
    def check_heptagon_collision(self, ball):
        center_x, center_y = WIDTH//2, HEIGHT//2
        
        # Check distance from center
        dx = ball.x - center_x
        dy = ball.y - center_y
        dist = math.sqrt(dx**2 + dy**2)
        
        if dist + ball.radius > self.heptagon.radius:
            # Find the normal vector from center to ball
            angle = math.atan2(dy, dx)
            normal_x = math.cos(angle)
            normal_y = math.sin(angle)
            
            # Move ball back inside heptagon
            overlap = (dist + ball.radius) - self.heptagon.radius
            ball.x -= overlap * normal_x
            ball.y -= overlap * normal_y
            
            # Reflect velocity
            dot_product = ball.vx * normal_x + ball.vy * normal_y
            ball.vx -= 2 * dot_product * normal_x * ELASTICITY
            ball.vy -= 2 * dot_product * normal_y * ELASTICITY
    
    def check_ball_collision(self, ball1, ball2):
        dx = ball2.x - ball1.x
        dy = ball2.y - ball1.y
        distance = math.sqrt(dx**2 + dy**2)
        return distance < (ball1.radius + ball2.radius)
    
    def resolve_ball_collision(self, ball1, ball2):
        dx = ball2.x - ball1.x
        dy = ball2.y - ball1.y
        distance = math.sqrt(dx**2 + dy**2)
        
        # Normal vector
        nx = dx / distance
        ny = dy / distance
        
        # Relative velocity
        dvx = ball2.vx - ball1.vx
        dvy = ball2.vy - ball1.vy
        
        # Calculate impulse
        impulse = 2 * (dvx * nx + dvy * ny) / 2
        impulse *= ELASTICITY
        
        # Apply impulse
        ball1.vx -= impulse * nx
        ball1.vy -= impulse * ny
        ball2.vx += impulse * nx
        ball2.vy += impulse * ny
        
        # Separate the balls to prevent sticking
        overlap = (ball1.radius + ball2.radius) - distance
        ball1.x -= overlap * nx / 2
        ball1.y -= overlap * ny / 2
        ball2.x += overlap * nx / 2
        ball2.y += overlap * ny / 2
    
    def run(self):
        self.root.mainloop()

if __name__ == "__main__":
    root = tk.Tk()
    root.title("Bouncing Balls in a Spinning Heptagon")
    app = BouncingBalls(root)
    app.run()
```

</details>

<details>

<summary>Float8 Heptagon code</summary>

```python
import tkinter as tk
import math
import numpy as np
from dataclasses import dataclass
from typing import List, Tuple, Optional

# Constants
WIDTH, HEIGHT = 800, 800
CENTER_X, CENTER_Y = WIDTH // 2, HEIGHT // 2
GRAVITY = 0.2
FRICTION = 0.99
BALL_RADIUS = 15
HEPTAGON_RADIUS = 300
ROTATION_SPEED = 360 / 5  # degrees per second
ELASTICITY = 0.8
SPIN_FRICTION = 0.98

# Colors for the balls
BALL_COLORS = [
    '#f8b862', '#f6ad49', '#f39800', '#f08300', '#ec6d51',
    '#ee7948', '#ed6d3d', '#ec6800', '#ec6800', '#ee7800',
    '#eb6238', '#ea5506', '#ea5506', '#eb6101', '#e49e61',
    '#e45e32', '#e17b34', '#dd7a56', '#db8449', '#d66a35'
]

@dataclass
class Ball:
    x: float
    y: float
    vx: float
    vy: float
    radius: float
    color: str
    number: int
    spin: float = 0.0

    def move(self):
        self.x += self.vx
        self.y += self.vy
        self.vy += GRAVITY
        self.vx *= FRICTION
        self.vy *= FRICTION
        self.spin *= SPIN_FRICTION

    def collide_with_ball(self, other: 'Ball'):
        dx = other.x - self.x
        dy = other.y - self.y
        distance = math.hypot(dx, dy)
        
        if distance < self.radius + other.radius:
            # Calculate collision normal
            nx = dx / distance
            ny = dy / distance
            
            # Calculate relative velocity
            dvx = other.vx - self.vx
            dvy = other.vy - self.vy
            
            # Calculate impulse
            impulse = 2 * (dvx * nx + dvy * ny) / (1/self.radius + 1/other.radius)
            
            # Apply impulse
            self.vx += impulse * nx / self.radius
            self.vy += impulse * ny / self.radius
            other.vx -= impulse * nx / other.radius
            other.vy -= impulse * ny / other.radius
            
            # Separate balls to prevent sticking
            overlap = (self.radius + other.radius - distance) / 2
            self.x -= overlap * nx
            self.y -= overlap * ny
            other.x += overlap * nx
            other.y += overlap * ny
            
            # Transfer some spin
            transfer = impulse * 0.01
            self.spin -= transfer
            other.spin += transfer

class HeptagonBounceSimulator:
    def __init__(self, root):
        self.root = root
        self.canvas = tk.Canvas(root, width=WIDTH, height=HEIGHT, bg='white')
        self.canvas.pack()
        
        self.balls = self.create_balls()
        self.heptagon_angle = 0
        self.last_time = 0
        self.running = True
        
        self.root.bind('<space>', self.toggle_pause)
        self.root.bind('<Escape>', lambda e: root.destroy())
        
        self.last_time = self.root.after(0, self.update)
    
    def create_balls(self) -> List[Ball]:
        balls = []
        for i in range(20):
            # Start all balls at center with small random velocity
            angle = np.random.uniform(0, 2 * math.pi)
            speed = np.random.uniform(0.5, 2)
            vx = math.cos(angle) * speed
            vy = math.sin(angle) * speed
            
            balls.append(Ball(
                x=CENTER_X,
                y=CENTER_Y,
                vx=vx,
                vy=vy,
                radius=BALL_RADIUS,
                color=BALL_COLORS[i],
                number=i+1,
                spin=np.random.uniform(-2, 2)
            ))
        return balls
    
    def toggle_pause(self, event):
        self.running = not self.running
        if self.running:
            self.last_time = self.root.after(0, self.update)
    
    def get_heptagon_vertices(self) -> List[Tuple[float, float]]:
        vertices = []
        for i in range(7):
            angle = math.radians(self.heptagon_angle + i * 360 / 7)
            x = CENTER_X + HEPTAGON_RADIUS * math.cos(angle)
            y = CENTER_Y + HEPTAGON_RADIUS * math.sin(angle)
            vertices.append((x, y))
        return vertices
    
    def check_ball_heptagon_collision(self, ball: Ball):
        vertices = self.get_heptagon_vertices()
        closest_dist = float('inf')
        closest_normal = (0, 0)
        closest_edge = None
        
        # Check collision with each edge of the heptagon
        for i in range(len(vertices)):
            p1 = vertices[i]
            p2 = vertices[(i + 1) % len(vertices)]
            
            # Vector from p1 to p2
            edge_x = p2[0] - p1[0]
            edge_y = p2[1] - p1[1]
            edge_length = math.hypot(edge_x, edge_y)
            
            # Normalize edge vector
            edge_x /= edge_length
            edge_y /= edge_length
            
            # Normal vector (perpendicular to edge, pointing inward)
            nx = -edge_y
            ny = edge_x
            
            # Vector from p1 to ball
            ball_to_p1_x = ball.x - p1[0]
            ball_to_p1_y = ball.y - p1[1]
            
            # Project ball onto edge normal
            projection = ball_to_p1_x * nx + ball_to_p1_y * ny
            
            # If projection is negative, ball is outside the heptagon
            if projection < ball.radius:
                # Find closest point on edge to ball
                edge_proj = ball_to_p1_x * edge_x + ball_to_p1_y * edge_y
                edge_proj = max(0, min(edge_length, edge_proj))
                closest_x = p1[0] + edge_proj * edge_x
                closest_y = p1[1] + edge_proj * edge_y
                
                # Distance from ball to closest point on edge
                dist = math.hypot(ball.x - closest_x, ball.y - closest_y)
                
                if dist < closest_dist:
                    closest_dist = dist
                    closest_normal = (nx, ny)
                    closest_edge = (p1, p2)
        
        if closest_dist < ball.radius:
            # Calculate bounce response
            dot_product = ball.vx * closest_normal[0] + ball.vy * closest_normal[1]
            
            # Apply bounce with elasticity
            ball.vx -= (1 + ELASTICITY) * dot_product * closest_normal[0]
            ball.vy -= (1 + ELASTICITY) * dot_product * closest_normal[1]
            
            # Add some spin based on impact
            edge_vec = (closest_edge[1][0] - closest_edge[0][0], 
                        closest_edge[1][1] - closest_edge[0][1])
            edge_length = math.hypot(edge_vec[0], edge_vec[1])
            if edge_length > 0:
                edge_vec = (edge_vec[0]/edge_length, edge_vec[1]/edge_length)
                # Cross product of velocity and edge direction
                spin_effect = (ball.vx * edge_vec[1] - ball.vy * edge_vec[0]) * 0.1
                ball.spin += spin_effect
            
            # Move ball outside the heptagon to prevent sticking
            penetration = ball.radius - closest_dist
            ball.x += penetration * closest_normal[0]
            ball.y += penetration * closest_normal[1]
    
    def update(self):
        if not self.running:
            return
        
        # Clear canvas
        self.canvas.delete('all')
        
        # Update heptagon rotation
        self.heptagon_angle += ROTATION_SPEED / 60  # Assuming ~60 FPS
        
        # Draw heptagon
        vertices = self.get_heptagon_vertices()
        self.canvas.create_polygon(vertices, outline='black', fill='', width=2)
        
        # Update and draw balls
        for i, ball in enumerate(self.balls):
            # Move ball
            ball.move()
            
            # Check collisions with heptagon
            self.check_ball_heptagon_collision(ball)
            
            # Draw ball
            self.canvas.create_oval(
                ball.x - ball.radius, ball.y - ball.radius,
                ball.x + ball.radius, ball.y + ball.radius,
                fill=ball.color, outline='black'
            )
            
            # Draw number with rotation based on spin
            angle = ball.spin * 10  # Scale spin for visible rotation
            self.canvas.create_text(
                ball.x, ball.y,
                text=str(ball.number),
                font=('Arial', 10, 'bold'),
                angle=angle
            )
        
        # Check ball-ball collisions
        for i in range(len(self.balls)):
            for j in range(i + 1, len(self.balls)):
                self.balls[i].collide_with_ball(self.balls[j])
        
        # Schedule next update
        self.last_time = self.root.after(16, self.update)  # ~60 FPS

if __name__ == '__main__':
    root = tk.Tk()
    root.title('Bouncing Balls in a Spinning Heptagon')
    simulator = HeptagonBounceSimulator(root)
    root.mainloop()
```

</details>

## :detective: Extra Findings & Tips

1. We find using lower KV cache quantization (4bit) seems to degrade generation quality via empirical tests - more tests need to be done, but we suggest using `q8_0` cache quantization. The goal of quantization is to support longer context lengths since the KV cache uses quite a bit of memory.
2. We found the `down_proj` in this model to be extremely sensitive to quantitation. We had to redo some of our dyanmic quants which used 2bits for `down_proj` and now we use 3bits as the minimum for all these matrices.
3. Using `llama.cpp` 's Flash Attention backend does result in somewhat faster decoding speeds. Use `-DGGML_CUDA_FA_ALL_QUANTS=ON` when compiling. Note it's also best to set your CUDA architecture as found in <https://developer.nvidia.com/cuda-gpus> to reduce compilation times, then set it via `-DCMAKE_CUDA_ARCHITECTURES="80"`
4. Using a `min_p=0.01`is probably enough. `llama.cpp`defaults to 0.1, which is probably not necessary. Since a temperature of 0.3 is used anyways, we most likely will very unlikely sample low probability tokens, so removing very unlikely tokens is a good idea. DeepSeek recommends 0.0 temperature for coding tasks.

[^1]: MUST USE 8bit - not 4bit

[^2]: CPU threads your machine has

[^3]: Approx 2 for 24GB GPU. Approx 18 for 80GB GPU.

[^4]: Context length


# DeepSeek-R1: How to Run Locally

A guide on how you can run our 1.58-bit Dynamic Quants for DeepSeek-R1 using llama.cpp.

{% hint style="success" %}
Please see <https://docs.unsloth.ai/basics/deepseek-r1-0528-how-to-run-locally> for an updated DeepSeek R1-0528 (May 28th 2025 version)
{% endhint %}

## Using llama.cpp (recommended)

1. Do not forget about `<｜User｜>` and `<｜Assistant｜>` tokens! - Or use a chat template formatter
2. Obtain the latest `llama.cpp` at: [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

3. It's best to use `--min-p 0.05` to counteract very rare token predictions - I found this to work well especially for the 1.58bit model.
4. Download the model via:

```python
# pip install huggingface_hub hf_transfer
# import os # Optional for faster downloading
# os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"

from huggingface_hub import snapshot_download
snapshot_download(
  repo_id = "unsloth/DeepSeek-R1-GGUF",
  local_dir = "DeepSeek-R1-GGUF",
  allow_patterns = ["*UD-IQ1_S*"], # Select quant type UD-IQ1_S for 1.58bit
)
```

6. Example with Q4\_0 K quantized cache **Notice -no-cnv disables auto conversation mode**

```bash
./llama.cpp/llama-cli \
    --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
    --cache-type-k q4_0 \
    -no-cnv --prio 2 \
    --temp 0.6 \
    --ctx-size 8192 \
    --seed 3407 \
    --prompt "<｜User｜>What is 1+1?<｜Assistant｜>"
```

Example output:

```txt
 <think>
 Okay, so I need to figure out what 1 plus 1 is. Hmm, where do I even start? I remember from school that adding numbers is pretty basic, but I want to make sure I understand it properly.
 Let me think, 1 plus 1. So, I have one item and I add another one. Maybe like a apple plus another apple. If I have one apple and someone gives me another, I now have two apples. So, 1 plus 1 should be 2. That makes sense.
 Wait, but sometimes math can be tricky. Could it be something else? Like, in a different number system maybe? But I think the question is straightforward, using regular numbers, not like binary or hexadecimal or anything.
 I also recall that in arithmetic, addition is combining quantities. So, if you have two quantities of 1, combining them gives you a total of 2. Yeah, that seems right.
 Is there a scenario where 1 plus 1 wouldn't be 2? I can't think of any...
```

4. If you have a GPU (RTX 4090 for example) with 24GB, you can offload multiple layers to the GPU for faster processing. If you have multiple GPUs, you can probably offload more layers.

```bash
./llama.cpp/llama-cli \
    --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
    --cache-type-k q4_0 \
    -no-cnv --prio 2 \
    --n-gpu-layers 7 \
    --temp 0.6 \
    --ctx-size 8192 \
    --seed 3407 \
    --prompt "<｜User｜>Create a Flappy Bird game in Python.<｜Assistant｜>"
```

5. To test our Flappy Bird example as mentioned in our blog post here: <https://unsloth.ai/blog/deepseekr1-dynamic>, we can produce the 2nd example like below using our 1.58bit dynamic quant:

<table data-column-title-hidden data-view="cards" data-full-width="false"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td>Original DeepSeek R1</td><td></td><td></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-3c484081174c631653c8c7bf7e7674f05255f740%2FInShot_20250127_043158375_H8Uu6tyJXYAFwUEIu04Am.gif?alt=media">InShot_20250127_043158375_H8Uu6tyJXYAFwUEIu04Am.gif</a></td></tr><tr><td>1.58bit Dynamic Quant</td><td></td><td></td><td><a href="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-c41eac0fea9362017e94123ee8f9793df21b8e97%2FInShot_20250127_042648160_lrtL8-eRhl4qtLaUDSU87.gif?alt=media">InShot_20250127_042648160_lrtL8-eRhl4qtLaUDSU87.gif</a></td></tr></tbody></table>

The prompt used is as below:

{% code overflow="wrap" %}

```
<｜User｜>Create a Flappy Bird game in Python. You must include these things:
1. You must use pygame.
2. The background color should be randomly chosen and is a light shade. Start with a light blue color.
3. Pressing SPACE multiple times will accelerate the bird.
4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.
5. Place on the bottom some land colored as dark brown or yellow chosen randomly.
6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.
7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.
8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.
The final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<｜Assistant｜>
```

{% endcode %}

To call llama.cpp using this example, we do:

```bash
./llama.cpp/llama-cli \
    --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
    --cache-type-k q4_0 \
    -no-cnv --prio 2 \
    --n-gpu-layers 7 \
    --temp 0.6 \
    --ctx-size 8192 \
    --seed 3407 \
    --prompt "<｜User｜>Create a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<｜Assistant｜>"
```

5. Also, if you want to merge the weights together for use in Ollama for example, use this script:

```bash
./llama.cpp/llama-gguf-split --merge \
    DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
    merged_file.gguf
```

6. DeepSeek R1 has 61 layers. For example with a 24GB GPU or 80GB GPU, you can expect to offload after rounding down (reduce by 1 if it goes out of memory):

| Quant   | File Size | 24GB GPU | 80GB GPU | 2x80GB GPU    |
| ------- | --------- | -------- | -------- | ------------- |
| 1.58bit | 131GB     | 7        | 33       | All layers 61 |
| 1.73bit | 158GB     | 5        | 26       | 57            |
| 2.22bit | 183GB     | 4        | 22       | 49            |
| 2.51bit | 212GB     | 2        | 19       | 32            |

### Running on Mac / Apple devices

For Apple Metal devices, be careful of --n-gpu-layers. If you find the machine going out of memory, reduce it. For a 128GB unified memory machine, you should be able to offload 59 layers or so.

```bash
./llama.cpp/llama-cli \
    --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
    --cache-type-k q4_0 \
    --prio 2 \
    --temp 0.6 \
    --ctx-size 8192 \
    --seed 3407 \
    --n-gpu-layers 59 \
    -no-cnv \
    --prompt "<｜User｜>Create a Flappy Bird game in Python.<｜Assistant｜>"
```

### Run in Ollama/Open WebUI

Open WebUI has made an step-by-step tutorial on how to run R1 here: <https://docs.openwebui.com/tutorials/integrations/llm-providers/deepseekr1-dynamic>

\
If you want to use Ollama for inference on GGUFs, you need to first merge the 3 GGUF split files into 1 like the code below. Then you will need to run the model locally.

```bash
./llama.cpp/llama-gguf-split --merge \
  DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
	merged_file.gguf
```

## DeepSeek Chat Template

All distilled versions and the main 671B R1 model use the same chat template:

`<｜begin▁of▁sentence｜><｜User｜>What is 1+1?<｜Assistant｜>It's 2.<｜end▁of▁sentence｜><｜User｜>Explain more!<｜Assistant｜>`

A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call *tokenizer.encode(..., add\_special\_tokens = False)* since the chat template auto adds a BOS token as well.\
For llama.cpp / GGUF inference, you should skip the BOS since it’ll auto add it.

`<｜User｜>What is 1+1?<｜Assistant｜>`

The \<think> and \</think> tokens get their own designated tokens. For the distilled versions for Qwen and Llama, some tokens are re-mapped, whilst Qwen for example did not have a BOS token, so <|object\_ref\_start|> had to be used instead.\
\
**Tokenizer ID Mappings:**

| Token                     | R1     | Distill Qwen | Distill Llama |
| ------------------------- | ------ | ------------ | ------------- |
| \<think>                  | 128798 | 151648       | 128013        |
| \</think>                 | 128799 | 151649       | 128014        |
| <\|begin\_of\_sentence\|> | 0      | 151646       | 128000        |
| <\|end\_of\_sentence\|>   | 1      | 151643       | 128001        |
| <\|User\|>                | 128803 | 151644       | 128011        |
| <\|Assistant\|>           | 128804 | 151645       | 128012        |
| Padding token             | 2      | 151654       | 128004        |

Original tokens in models:

| Token                 | Qwen 2.5 32B Base        | Llama 3.3 70B Instruct            |
| --------------------- | ------------------------ | --------------------------------- |
| \<think>              | <\|box\_start\|>         | <\|reserved\_special\_token\_5\|> |
| \</think>             | <\|box\_end\|>           | <\|reserved\_special\_token\_6\|> |
| <｜begin▁of▁sentence｜> | <\|object\_ref\_start\|> | <\|begin\_of\_text\|>             |
| <｜end▁of▁sentence｜>   | <\|endoftext\|>          | <\|end\_of\_text\|>               |
| <｜User｜>              | <\|im\_start\|>          | <\|reserved\_special\_token\_3\|> |
| <｜Assistant｜>         | <\|im\_end\|>            | <\|reserved\_special\_token\_4\|> |
| Padding token         | <\|vision\_pad\|>        | <\|finetune\_right\_pad\_id\|>    |

All Distilled and the original R1 versions seem to have accidentally assigned the padding token to <｜end▁of▁sentence｜>, which is mostly not a good idea, especially if you want to further finetune on top of these reasoning models. This will cause endless infinite generations, since most frameworks will mask the EOS token out as -100.\
\
We fixed all distilled and the original R1 versions with the correct padding token (Qwen uses <|vision\_pad|>, Llama uses <|finetune\_right\_pad\_id|>, and R1 uses <｜▁pad▁｜> or our own added <｜PAD▁TOKEN｜>.

## GGUF R1 Table

<table data-full-width="true"><thead><tr><th>MoE Bits</th><th>Type</th><th>Disk Size</th><th>Accuracy</th><th>Link</th><th>Details</th></tr></thead><tbody><tr><td>1.58bit</td><td>UD-IQ1_S</td><td><strong>131GB</strong></td><td>Fair</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ1_S">Link</a></td><td>MoE all 1.56bit. <code>down_proj</code> in MoE mixture of 2.06/1.56bit</td></tr><tr><td>1.73bit</td><td>UD-IQ1_M</td><td><strong>158GB</strong></td><td>Good</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ1_M">Link</a></td><td>MoE all 1.56bit. <code>down_proj</code> in MoE left at 2.06bit</td></tr><tr><td>2.22bit</td><td>UD-IQ2_XXS</td><td><strong>183GB</strong></td><td>Better</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ2_XXS">Link</a></td><td>MoE all 2.06bit. <code>down_proj</code> in MoE mixture of 2.5/2.06bit</td></tr><tr><td>2.51bit</td><td>UD-Q2_K_XL</td><td><strong>212GB</strong></td><td>Best</td><td><a href="https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-Q2_K_XL">Link</a></td><td>MoE all 2.5bit. <code>down_proj</code> in MoE mixture of 3.5/2.5bit</td></tr></tbody></table>


# DeepSeek-R1 Dynamic 1.58-bit

See performance comparison tables for Unsloth's Dynamic GGUF Quants vs Standard IMatrix Quants.

Read our full DeepSeek-R1 blogpost here: [unsloth.ai/blog/deepseekr1-dynamic](https://unsloth.ai/blog/deepseekr1-dynamic)

### 1-bit (Small) - Dynamic vs. Basic

<table data-full-width="true"><thead><tr><th>GGUF Type</th><th>Quant</th><th>Size (GB)</th><th>Seed</th><th>Pygame</th><th>Background</th><th>Accelerate SPACE</th><th>Bird shape</th><th>Land</th><th>Top right score</th><th>Pipes</th><th>Best Score</th><th>Quit</th><th>Runnable</th><th>Score</th><th>Avg Score</th><th width="214">Errors</th><th width="421">Notes</th></tr></thead><tbody><tr><td>Dynamic</td><td>IQ1_S</td><td>131</td><td>3407</td><td>1</td><td>0.5</td><td>1</td><td>0.5</td><td>0.5</td><td>1</td><td>0.5</td><td>1</td><td>1</td><td>0</td><td>7</td><td></td><td>score =!inc SyntaxError: invalid syntax</td><td>Selects random shapes and colors at the start, but doesn't rotate across trials</td></tr><tr><td>Dynamic</td><td>IQ1_S</td><td>131</td><td>3408</td><td>1</td><td>1</td><td>0.25</td><td>1</td><td>0.5</td><td>1</td><td>0.5</td><td>1</td><td>1</td><td>0</td><td>7.25</td><td></td><td>score =B4 NameError: name 'B4' is not defined</td><td>Better - selects pipe colors randomnly, but all are just 1 color - should be different. Dropping to ground fails to reset acceleration.</td></tr><tr><td>Dynamic</td><td>IQ1_S</td><td>131</td><td>3409</td><td>1</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>6.5</td><td>6.92</td><td>score =3D 0 SyntaxError: invalid decimal literal</td><td>Too hard to play - acceleration too fast. Pipe colors now are random, but bird shape not changing. Land collison fails.</td></tr><tr><td>Basic</td><td>IQ1_S</td><td>133</td><td>3407</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td><td>No code</td><td>Fully failed. Repeats "with Dark Colurs" forever</td></tr><tr><td>Basic</td><td>IQ1_S</td><td>133</td><td>3408</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td></td><td>No code</td><td>Fully failed. Repeats "Pygame's" forever</td></tr><tr><td>Basic</td><td>IQ1_S</td><td>133</td><td>3409</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>No code</td><td>Fully failed. Repeats "pipe_x = screen_height<br>pipe_x = screen_height<br>pipe_height = screen_height - Pipe_height" forever.</td></tr></tbody></table>

### 1-bit (Medium) - Dynamic vs. Basic

<table data-full-width="true"><thead><tr><th>GGUF Type</th><th>Quant</th><th>Size (GB)</th><th>Seed</th><th>Pygame</th><th>Background</th><th>Accelerate SPACE</th><th>Bird shape</th><th>Land</th><th>Top right score</th><th>Pipes</th><th>Best Score</th><th>Quit</th><th>Runnable</th><th>Score</th><th>Avg Score</th><th width="268">Errors</th><th width="284">Notes</th></tr></thead><tbody><tr><td>Dynamic</td><td>IQ1_M</td><td>158</td><td>3407</td><td>1</td><td>1</td><td>0.75</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>9.75</td><td></td><td>None</td><td>A bit fast and hard to play.</td></tr><tr><td>Dynamic</td><td>IQ1_M</td><td>158</td><td>3408</td><td>1</td><td>1</td><td>0.5</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>9.5</td><td></td><td>None</td><td>Very good - land should be clearer. Acceleration should be slower.</td></tr><tr><td>Dynamic</td><td>IQ1_M</td><td>158</td><td>3409</td><td>1</td><td>0.5</td><td>1</td><td>0.5</td><td>0.5</td><td>1</td><td>0.5</td><td>1</td><td>1</td><td>1</td><td>8</td><td>9.08</td><td>None</td><td>Background color does not change across trials.Pipes do not touch the top. No land is seen.</td></tr><tr><td>Basic</td><td>IQ1_M</td><td>149</td><td>3407</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>2</td><td></td><td>if game_over: NameError: name 'game_over' is not defined</td><td>Fully failed. Black screen only</td></tr><tr><td>Basic</td><td>IQ1_M</td><td>149</td><td>3408</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>2</td><td></td><td>No code</td><td>Fully failed. Black screen then closes.</td></tr><tr><td>Basic</td><td>IQ1_M</td><td>149</td><td>3409</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1.67</td><td>window.fill((100, 100, 255)) Light Blue SyntaxError: invalid syntax &#x26;&#x26; main() NameError: name 'main' is not defined.</td><td>Fully failed.</td></tr></tbody></table>

### 2-bit (Extra extra Small) - Dynamic vs. Basic

<table data-full-width="true"><thead><tr><th>GGUF Type</th><th>Quant</th><th>Size (GB)</th><th>Seed</th><th>Pygame</th><th>Background</th><th>Accelerate SPACE</th><th>Bird shape</th><th>Land</th><th>Top right score</th><th>Pipes</th><th>Best Score</th><th>Quit</th><th>Runnable</th><th>Score</th><th>Avg Score</th><th width="330">Errors</th><th width="260">Notes</th><th></th></tr></thead><tbody><tr><td>Dynamic</td><td>IQ2_XXS</td><td>183</td><td>3407</td><td>1</td><td>1</td><td>0.5</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>9.5</td><td></td><td>None</td><td>Too hard to play - acceleration too slow. Lags</td><td></td></tr><tr><td>Dynamic</td><td>IQ2_XXS</td><td>183</td><td>3408</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0.5</td><td>0.5</td><td>1</td><td>0</td><td>8</td><td></td><td>global best_score SyntaxError: name 'best_score' is assigned to before global declaration</td><td>Had to edit 2 lines - remove global best_score, and set pipe_list = []</td><td></td></tr><tr><td>Dynamic</td><td>IQ2_XXS</td><td>183</td><td>3409</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>10</td><td>9.17</td><td>None</td><td>Extremely good. Even makes pipes have random distances between them.</td><td></td></tr><tr><td>Basic</td><td>IQ2_XXS</td><td>175</td><td>3407</td><td>1</td><td>0.5</td><td>0.5</td><td>0.5</td><td>1</td><td>0</td><td>0.5</td><td>1</td><td>0</td><td>0</td><td>5</td><td></td><td>pipe_color = random.choice([(34, 139, 34), (139, 69, 19), (47, 47, 47)) SyntaxError: closing parenthesis ')' does not match opening parenthesis '[' &#x26;&#x26; pygame.draw.polygon(screen, bird_color, points) ValueError: points argument must contain more than 2 points</td><td>Fails quiting. Same color. Collison detection a bit off. No score</td><td></td></tr><tr><td>Basic</td><td>IQ2_XXS</td><td>175</td><td>3408</td><td>1</td><td>0.5</td><td>0.5</td><td>0.5</td><td>1</td><td>1</td><td>0.5</td><td>1</td><td>0</td><td>0</td><td>6</td><td></td><td>pipes.append({'x': SCREEN_WIDTH, 'gap_y': random.randint(50, SCREEN_HEIGHT - 150)) SyntaxError: closing parenthesis ')' does not match opening parenthesis '{'</td><td>Acceleration weird. Chooses 1 color per round. Cannot quit.</td><td></td></tr><tr><td>Basic</td><td>IQ2_XXS</td><td>175</td><td>3409</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0.5</td><td>0</td><td>7.5</td><td>6.17</td><td>screen = pygame.display.set_mode((SCREEN_WIDTH, SCREENHEIGHT)) NameError: name 'SCREENHEIGHT' is not defined. Did you mean: 'SCREEN_HEIGHT'?</td><td>OK. Colors change. Best score does not update. Quit only ESC not Q.</td><td></td></tr></tbody></table>

### **Dynamic Quantization trial output**

{% tabs %}
{% tab title="IQ1\_S code" %}
{% file src="/files/lhdwLrRjoyxL2ldYhPxu" %}

{% file src="/files/DtUWlLcTKdSCfUbbtgZR" %}

{% file src="/files/DDNKmAaalnlnEM5jRbg8" %}
{% endtab %}

{% tab title="IQ1\_M code" %}
{% file src="/files/hHRiMbIGdY46iwSTPe1b" %}

{% file src="/files/Rtx9xWIBjjKRrrzu2CG2" %}

{% file src="/files/8JEm5DHJCgA88hsCoFhi" %}
{% endtab %}

{% tab title="IQ2\_XXS code" %}
{% file src="/files/mz4oALXadBvCtHguz3yp" %}

{% file src="/files/0qtRS8oIwuPzSSEyltfz" %}

{% file src="/files/etx6UqoSp06eUA1QFJLB" %}
{% endtab %}
{% endtabs %}

### Non Dynamic Quantization trial output

{% tabs %}
{% tab title="IQ1\_S basic code" %}
{% file src="/files/w1ns6l9VftZdYwuzJknB" %}

{% file src="/files/blG94CHcCAzXomVvOjJm" %}

{% file src="/files/3Lvf4jH7b9WDlGZXoSLZ" %}
{% endtab %}

{% tab title="IQ1\_M basic code" %}
{% file src="/files/rtT6jQe7SDuap1diCINq" %}

{% file src="/files/HCqDRClY6WfcsyEuq9LG" %}

{% file src="/files/Oy2Qj1vRPqENGfqhMvwp" %}
{% endtab %}

{% tab title="IQ2\_XXS basic code" %}
{% file src="/files/UY2CMIdGxFSuv1SzUPFm" %}

{% file src="/files/jvMphveTauL4A2wUyziV" %}

{% file src="/files/nJIxMdGX3Yv60kIFyFHW" %}
{% endtab %}
{% endtabs %}


# Phi-4 Reasoning: How to Run & Fine-tune

Learn to run & fine-tune Phi-4 reasoning models locally with Unsloth + our Dynamic 2.0 quants

Microsoft's new Phi-4 reasoning models are now supported in Unsloth. The 'plus' variant performs on par with OpenAI's o1-mini, o3-mini and Sonnet 3.7. The 'plus' and standard reasoning models are 14B parameters while the 'mini' has 4B parameters.\
\
All Phi-4 reasoning uploads use our [Unsloth Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) methodology.

#### **Phi-4 reasoning - Unsloth Dynamic 2.0 uploads:**

| Dynamic 2.0 GGUF (to run)                                                                                                                                                                                                                                                                                    | Dynamic 4-bit Safetensor (to finetune/deploy)                                                                                                                                                                                                                                                                                 |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <ul><li><a href="https://huggingface.co/unsloth/Phi-4-reasoning-plus-GGUF/">Reasoning-plus</a> (14B)</li><li><a href="https://huggingface.co/unsloth/Phi-4-reasoning-GGUF">Reasoning</a> (14B)</li><li><a href="https://huggingface.co/unsloth/Phi-4-mini-reasoning-GGUF/">Mini-reasoning</a> (4B)</li></ul> | <ul><li><a href="https://huggingface.co/unsloth/Phi-4-reasoning-plus-unsloth-bnb-4bit">Reasoning-plus</a></li><li><a href="https://huggingface.co/unsloth/phi-4-reasoning-unsloth-bnb-4bit">Reasoning</a></li><li><a href="https://huggingface.co/unsloth/Phi-4-mini-reasoning-unsloth-bnb-4bit">Mini-reasoning</a></li></ul> |

## 🖥️ **Running Phi-4 reasoning**

### :gear: Official Recommended Settings

According to Microsoft, these are the recommended settings for inference:

* <mark style="background-color:blue;">**Temperature = 0.8**</mark>
* Top\_P = 0.95

### **Phi-4 reasoning Chat templates**

Please ensure you use the correct chat template as the 'mini' variant has a different one.

#### **Phi-4-mini:**

{% code overflow="wrap" %}

```
<|system|>Your name is Phi, an AI math expert developed by Microsoft.<|end|><|user|>How to solve 3*x^2+4*x+5=1?<|end|><|assistant|>
```

{% endcode %}

#### **Phi-4-reasoning and Phi-4-reasoning-plus:**

This format is used for general conversation and instructions:

{% code overflow="wrap" %}

```
<|im_start|>system<|im_sep|>You are Phi, a language model trained by Microsoft to help users. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> {Thought section} </think> {Solution section}. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion. Now, try to solve the following question through the above guidelines:<|im_end|><|im_start|>user<|im_sep|>What is 1+1?<|im_end|><|im_start|>assistant<|im_sep|>
```

{% endcode %}

{% hint style="info" %}
Yes, the chat template/prompt format is this long!
{% endhint %}

### 🦙 Ollama: Run Phi-4 reasoning Tutorial

1. Install `ollama` if you haven't already!

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run the model! Note you can call `ollama serve`in another terminal if it fails. We include all our fixes and suggested parameters (temperature etc) in `params` in our Hugging Face upload.

```bash
ollama run hf.co/unsloth/Phi-4-mini-reasoning-GGUF:Q4_K_XL
```

### 📖 Llama.cpp: Run Phi-4 reasoning Tutorial

{% hint style="warning" %}
You must use `--jinja` in llama.cpp to enable reasoning for the models, expect for the 'mini' variant. Otherwise no token will be provided.
{% endhint %}

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Phi-4-mini-reasoning-GGUF",
    local_dir = "unsloth/Phi-4-mini-reasoning-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)
```

3. Run the model in conversational mode in llama.cpp. You must use `--jinja` in llama.cpp to enable reasoning for the models. This is however not needed if you're using the 'mini' variant.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Phi-4-mini-reasoning-GGUF/Phi-4-mini-reasoning-UD-Q4_K_XL.gguf \
    --threads -1 \
    --n-gpu-layers 99 \
    --prio 3 \
    --temp 0.8 \
    --top-p 0.95 \
    --jinja \
    --min-p 0.00 \
    --ctx-size 32768 \
    --seed 3407
```

## 🦥 Fine-tuning Phi-4 with Unsloth

[Phi-4 fine-tuning](https://unsloth.ai/blog/phi4) for the models are also now supported in Unsloth. To fine-tune for free on Google Colab, just change the `model_name` of 'unsloth/Phi-4' to 'unsloth/Phi-4-mini-reasoning' etc.

* [Phi-4 (14B) fine-tuning notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb)


# QwQ-32B: How to Run effectively

How to run QwQ-32B effectively with our bug fixes and without endless generations + GGUFs.

Qwen released QwQ-32B - a reasoning model with performance comparable to DeepSeek-R1 on many [benchmarks](https://qwenlm.github.io/blog/qwq-32b/). However, people have been experiencing **infinite generations**, **many repetitions**, \<think> token issues and finetuning issues. We hope this guide will help debug and fix most issues!

{% hint style="info" %}
Our model uploads with our bug fixes work great for fine-tuning, vLLM and Transformers. If you're using llama.cpp and engines that use llama.cpp as backend, follow our [instructions here](#tutorial-how-to-run-qwq-32b) to fix endless generations.
{% endhint %}

**Unsloth QwQ-32B uploads with our bug fixes:**

| [GGUF](https://huggingface.co/unsloth/QwQ-32B-GGUF) | [Dynamic 4-bit](https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit) | [BnB 4-bit](https://huggingface.co/unsloth/QwQ-32B-bnb-4bit) | [16-bit](https://huggingface.co/unsloth/QwQ-32B) |
| --------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------ |

## :gear: Official Recommended Settings

According to [Qwen](https://huggingface.co/Qwen/QwQ-32B), these are the recommended settings for inference:

* Temperature of 0.6
* Top\_K of 40 (or 20 to 40)
* Min\_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
* Top\_P of 0.95
* Repetition Penalty of 1.0. (1.0 means disabled in llama.cpp and transformers)
* Chat template: `<|im_start|>user\nCreate a Flappy Bird game in Python.<|im_end|>\n<|im_start|>assistant\n<think>\n`

{% hint style="warning" %}
`llama.cpp` uses `min_p = 0.1`by default, which might cause issues. Force it to 0.0.
{% endhint %}

## :thumbsup: Recommended settings for llama.cpp

We noticed many people use a `Repetition Penalty` greater than 1.0. For example 1.1 to 1.5. This actually interferes with llama.cpp's sampling mechanisms. The goal of a repetition penalty is to penalize repeated generations, but we found this doesn't work as expected.

Turning off `Repetition Penalty` also works (ie setting it to 1.0), but we found using it to be useful to penalize endless generations.

To use it, we found you must also edit the ordering of samplers in llama.cpp to before applying `Repetition Penalty`, otherwise there will be endless generations. So add this:

```bash
--samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc"
```

By default, llama.cpp uses this ordering:

```bash
--samplers "dry;top_k;typ_p;top_p;min_p;xtc;temperature"
```

We reorder essentially temperature and dry, and move min\_p forward. This means we apply samplers in this order:

```bash
top_k=40
top_p=0.95
min_p=0.0
temperature=0.6
dry
typ_p
xtc
```

If you still encounter issues, you can increase the`--repeat-penalty 1.0 to 1.2 or 1.3.`

Courtesy to [@krist486](https://x.com/krist486/status/1897885598196654180) for bringing llama.cpp sampling directions to my attention.

## :sunny: Dry Repetition Penalty

We investigated usage of `dry penalty` as suggested in <https://github.com/ggml-org/llama.cpp/blob/master/examples/main/README.md> using a value of 0.8, but we actually found this to **rather cause syntax issues especially for coding**. If you still encounter issues, you can increase the`dry penalty to 0.8.`

Utilizing our swapped sampling ordering can also help if you decide to use `dry penalty`.

## :llama: Tutorial: How to Run QwQ-32B in Ollama

1. Install `ollama` if you haven't already!

```bash
apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh
```

2. Run run the model! Note you can call `ollama serve`in another terminal if it fails! We include all our fixes and suggested parameters (temperature, min\_p etc) in `param` in our Hugging Face upload!

```bash
ollama run hf.co/unsloth/QwQ-32B-GGUF:Q4_K_M
```

## 📖 Tutorial: How to Run QwQ-32B in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

2. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose Q4\_K\_M, or other quantized versions (like BF16 full precision). More versions at: <https://huggingface.co/unsloth/QwQ-32B-GGUF>

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/QwQ-32B-GGUF",
    local_dir = "unsloth-QwQ-32B-GGUF",
    allow_patterns = ["*Q4_K_M*"], # For Q4_K_M
)
```

3. Run Unsloth's Flappy Bird test, which will save the output to `Q4_K_M_yes_samplers.txt`
4. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 99` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
5. We use `--repeat-penalty 1.1` and `--dry-multiplier 0.5` which you can adjust.

```bash
./llama.cpp/llama-cli \
    --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \
    --threads 32 \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --repeat-penalty 1.1 \
    --dry-multiplier 0.5 \
    --min-p 0.01 \
    --top-k 40 \
    --top-p 0.95 \
    -no-cnv \
    --samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc" \
    --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n<think>\n"  \
        2>&1 | tee Q4_K_M_yes_samplers.txt
```

The full input from our <https://unsloth.ai/blog/deepseekr1-dynamic> 1.58bit blog is:

```
<|im_start|>user
Create a Flappy Bird game in Python. You must include these things:
1. You must use pygame.
2. The background color should be randomly chosen and is a light shade. Start with a light blue color.
3. Pressing SPACE multiple times will accelerate the bird.
4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.
5. Place on the bottom some land colored as dark brown or yellow chosen randomly.
6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.
7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.
8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.
The final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>
<|im_start|>assistant
<think>
```

The beginning and the end of the final Python output after removing the thinking parts:

```python
import pygame
import random
import sys

pygame.init()
### Continues

class Bird:
    def __init__(self):
        ### Continues

def main():
    best_score = 0
    current_score = 0
    game_over = False
    pipes = []
    first_time = True  # Track first game play

    # Initial setup
    background_color = (173, 216, 230)  # Light blue initially
    land_color = random.choice(land_colors)
    bird = Bird()

    while True:
        for event in pygame.event.get():
            ### Continues

        if not game_over:
            # Update bird and pipes
            bird.update()
            ### Continues

        # Drawing
        ### Continues
        pygame.display.flip()
        clock.tick(60)

if __name__ == "__main__":
    main()
```

<details>

<summary>Full final Python output (removed thinking parts):</summary>

{% code fullWidth="false" %}

```python
import pygame
import random
import sys

pygame.init()
WIDTH, HEIGHT = 800, 600
screen = pygame.display.set_mode((WIDTH, HEIGHT))
pygame.display.set_caption("Flappy Bird Clone")
clock = pygame.time.Clock()

# Colors
pipe_colors = [(0, 100, 0), (210, 180, 140), (50, 50, 50)]
land_colors = [(139, 69, 19), (255, 255, 0)]

# Game constants
GRAVITY = 0.5
PIPE_SPEED = 5
BIRD_SIZE = 30
LAND_HEIGHT = 50
PIPE_WIDTH = 50
PIPE_GAP = 150

class Bird:
    def __init__(self):
        self.x = WIDTH // 2
        self.y = HEIGHT // 2
        self.velocity = 0
        self.shape = random.choice(['square', 'circle', 'triangle'])
        self.color = (random.randint(0, 100), random.randint(0, 100), random.randint(0, 100))
        self.rect = pygame.Rect(self.x - BIRD_SIZE//2, self.y - BIRD_SIZE//2, BIRD_SIZE, BIRD_SIZE)
    
    def update(self):
        self.velocity += GRAVITY
        self.y += self.velocity
        self.rect.y = self.y - BIRD_SIZE//2
        self.rect.x = self.x - BIRD_SIZE//2  # Keep x centered
    
    def draw(self):
        if self.shape == 'square':
            pygame.draw.rect(screen, self.color, self.rect)
        elif self.shape == 'circle':
            pygame.draw.circle(screen, self.color, (self.rect.centerx, self.rect.centery), BIRD_SIZE//2)
        elif self.shape == 'triangle':
            points = [
                (self.rect.centerx, self.rect.top),
                (self.rect.left, self.rect.bottom),
                (self.rect.right, self.rect.bottom)
            ]
            pygame.draw.polygon(screen, self.color, points)

def spawn_pipe():
    pipe_x = WIDTH
    top_height = random.randint(50, HEIGHT - PIPE_GAP - LAND_HEIGHT)
    rect_top = pygame.Rect(pipe_x, 0, PIPE_WIDTH, top_height)
    bottom_y = top_height + PIPE_GAP
    bottom_height = (HEIGHT - LAND_HEIGHT) - bottom_y
    rect_bottom = pygame.Rect(pipe_x, bottom_y, PIPE_WIDTH, bottom_height)
    color = random.choice(pipe_colors)
    return {
        'rect_top': rect_top,
        'rect_bottom': rect_bottom,
        'color': color,
        'scored': False
    }

def main():
    best_score = 0
    current_score = 0
    game_over = False
    pipes = []
    first_time = True  # Track first game play

    # Initial setup
    background_color = (173, 216, 230)  # Light blue initially
    land_color = random.choice(land_colors)
    bird = Bird()

    while True:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                pygame.quit()
                sys.exit()
            if event.type == pygame.KEYDOWN:
                if event.key == pygame.K_ESCAPE or event.key == pygame.K_q:
                    pygame.quit()
                    sys.exit()
                if event.key == pygame.K_SPACE:
                    if game_over:
                        # Reset the game
                        bird = Bird()
                        pipes.clear()
                        current_score = 0
                        if first_time:
                            # First restart after initial game over
                            background_color = (random.randint(200, 255), random.randint(200, 255), random.randint(200, 255))
                            first_time = False
                        else:
                            background_color = (random.randint(200, 255), random.randint(200, 255), random.randint(200, 255))
                        land_color = random.choice(land_colors)
                        game_over = False
                    else:
                        # Jump the bird
                        bird.velocity = -15  # Initial upward velocity

        if not game_over:
            # Update bird and pipes
            bird.update()

            # Move pipes left
            remove_pipes = []
            for pipe in pipes:
                pipe['rect_top'].x -= PIPE_SPEED
                pipe['rect_bottom'].x -= PIPE_SPEED
                # Check if bird passed the pipe
                if not pipe['scored'] and bird.rect.x > pipe['rect_top'].right:
                    current_score += 1
                    pipe['scored'] = True
                # Check if pipe is offscreen
                if pipe['rect_top'].right < 0:
                    remove_pipes.append(pipe)
            # Remove offscreen pipes
            for p in remove_pipes:
                pipes.remove(p)

            # Spawn new pipe if needed
            if not pipes or pipes[-1]['rect_top'].x < WIDTH - 200:
                pipes.append(spawn_pipe())

            # Check collisions
            land_rect = pygame.Rect(0, HEIGHT - LAND_HEIGHT, WIDTH, LAND_HEIGHT)
            bird_rect = bird.rect
            # Check pipes
            for pipe in pipes:
                if bird_rect.colliderect(pipe['rect_top']) or bird_rect.colliderect(pipe['rect_bottom']):
                    game_over = True
                    break
            # Check land and top
            if bird_rect.bottom >= land_rect.top or bird_rect.top <= 0:
                game_over = True

            if game_over:
                if current_score > best_score:
                    best_score = current_score

        # Drawing
        screen.fill(background_color)
        # Draw pipes
        for pipe in pipes:
            pygame.draw.rect(screen, pipe['color'], pipe['rect_top'])
            pygame.draw.rect(screen, pipe['color'], pipe['rect_bottom'])
        # Draw land
        pygame.draw.rect(screen, land_color, (0, HEIGHT - LAND_HEIGHT, WIDTH, LAND_HEIGHT))
        # Draw bird
        bird.draw()
        # Draw score
        font = pygame.font.SysFont(None, 36)
        score_text = font.render(f'Score: {current_score}', True, (0, 0, 0))
        screen.blit(score_text, (WIDTH - 150, 10))
        # Game over screen
        if game_over:
            over_text = font.render('Game Over!', True, (255, 0, 0))
            best_text = font.render(f'Best: {best_score}', True, (255, 0, 0))
            restart_text = font.render('Press SPACE to restart', True, (255, 0, 0))
            screen.blit(over_text, (WIDTH//2 - 70, HEIGHT//2 - 30))
            screen.blit(best_text, (WIDTH//2 - 50, HEIGHT//2 + 10))
            screen.blit(restart_text, (WIDTH//2 - 100, HEIGHT//2 + 50))
        
        pygame.display.flip()
        clock.tick(60)

if __name__ == "__main__":
    main()
```

{% endcode %}

</details>

6. When running it, we get a runnable game!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-1740ecd246c7f62c363bc0ba1e0f1099b2f9b2a4%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

7. Now try the same without our fixes! So remove `--samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc"` This will save the output to `Q4_K_M_no_samplers.txt`

```bash
./llama.cpp/llama-cli \
    --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \
    --threads 32 \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --repeat-penalty 1.1 \
    --dry-multiplier 0.5 \
    --min-p 0.01 \
    --top-k 40 \
    --top-p 0.95 \
    -no-cnv \
    --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n<think>\n"  \
        2>&1 | tee Q4_K_M_no_samplers.txt
```

You will get some looping, but **problematically incorrect Python syntax** and many other issues. For example the below looks correct, but is wrong! Ie line 39 `pipes.clear() ### <<< NameError: name 'pipes' is not defined. Did you forget to import 'pipes'?`

{% code overflow="wrap" lineNumbers="true" %}

```python
import pygame
import random

pygame.init()

# Constants
WIDTH, HEIGHT = 800, 600
GROUND_HEIGHT = 20
GRAVITY = 0.7
PIPE_SPEED = -3
BIRD_SIZE = 45
MIN_GAP = 130
MAX_GAP = 200
PIPE_COLORS = [(0, 96, 0), (205, 133, 63), (89, 97, 107)]
DARK_BROWN = (94, 72, 4)
YELLOW = (252, 228, 6)

screen = pygame.display.set_mode((WIDTH, HEIGHT))
clock = pygame.time.Clock()

def random_light_color():
    return (
        random.randint(180, 230),
        random.randint(190, 300),
        random.randint(250, 255)
    )

def reset_game():
    global bird_x, bird_y
    global pipes, score
    global background_color, land_color
    global bird_shape, bird_color

    # Bird properties
    bird_x = WIDTH * 0.3
    bird_y = HEIGHT // 2
    bird_vel = -5  # Initial upward thrust

    pipes.clear() ### <<< NameError: name 'pipes' is not defined. Did you forget to import 'pipes'?
```

{% endcode %}

8. If you use `--repeat-penalty 1.5`, it gets even worse and more obvious, with actually totally incorrect syntax.

```python
import pygame
from random import randint  # For generating colors/shapes/positions randomly 
pygame.init()

# Constants:
WIDTH, HEIGHT =456 ,702   #
BACKGROUND_COLOR_LIGHTS=['lightskyblue']
GAP_SIZE=189           #

BIRD_RADIUS=3.  
PIPE_SPEED=- ( )    ? 
class Game():
def __init__(self):
        self.screen_size=( )

def reset_game_vars():
    global current_scor e
   # set to zero and other initial states.

# Main game loop:
while running :
     for event in pygame.event.get() : 
        if quit ... etc

pygame.quit()
print("Code is simplified. Due time constraints, full working version requires further implementation.")
```

9. You might be wondering maybe it's Q4\_K\_M? B16 ie full precision should work fine right? Incorrect - the outputs again fail if we do not use our fix of -`-samplers "top_k;top_p;min_p;temperature;dry;typ_p;xtc"` when using a Repetition Penalty.

## :sunrise\_over\_mountains: Still doesn't work? Try Min\_p = 0.1, Temperature = 1.5

According to the Min\_p paper <https://arxiv.org/pdf/2407.01082>, for more creative and diverse outputs, and if you still see repetitions, try disabling top\_p and top\_k!

```bash
./llama.cpp/llama-cli --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \
    --threads 32 --n-gpu-layers 99 \
    --ctx-size 16384 \
    --temp 1.5 \
    --min-p 0.1 \
    --top-k 0 \
    --top-p 1.0 \
    -no-cnv \
    --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n<think>\n"
```

Another approach is to disable `min_p` directly, since llama.cpp by default uses `min_p = 0.1`!

```bash
./llama.cpp/llama-cli --model unsloth-QwQ-32B-GGUF/QwQ-32B-Q4_K_M.gguf \
    --threads 32 --n-gpu-layers 99 \
    --ctx-size 16384 \
    --temp 0.6 \
    --min-p 0.0 \
    --top-k 40 \
    --top-p 0.95 \
    -no-cnv \
    --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n<think>\n"
```

## :thinking: \<think> token not shown?

Some people are reporting that because \<think> is default added in the chat template, some systems are not outputting the thinking traces correctly. You will have to manually edit the Jinja template from:

{% code overflow="wrap" %}

```
```

{% endcode %}

to another by removing the `<think>\n` at the end. The model will now have to manually add `<think>\n` during inference, which might not always succeed. DeepSeek also edited all models to default add a `<think>` token to force the model to go into reasoning model.

So change `{%- if add_generation_prompt %} {{- '<|im_start|>assistant\n<think>\n' }} {%- endif %}` to `{%- if add_generation_prompt %} {{- '<|im_start|>assistant\n' }} {%- endif %}`

ie remove `<think>\n`

<details>

<summary>Full jinja template with removed &#x3C;think>\n part</summary>

{% code overflow="wrap" %}

```
```

{% endcode %}

</details>

## Extra Notes

We first thought maybe:

1. QwQ's context length was not natively 128K, but rather 32K with YaRN extension. For example in the readme file for <https://huggingface.co/Qwen/QwQ-32B>, we see:

```json
{
  ...,
  "rope_scaling": {
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "type": "yarn"
  }
}
```

We tried overriding llama.cpp's YaRN handling, but nothing changed.

{% code overflow="wrap" %}

```bash
--override-kv qwen2.context_length=int:131072 \
--override-kv qwen2.rope.scaling.type=str:yarn \
--override-kv qwen2.rope.scaling.factor=float:4 \
--override-kv qwen2.rope.scaling.original_context_length=int:32768 \
--override-kv qwen2.rope.scaling.attn_factor=float:1.13862943649292 \
```

{% endcode %}

2. We also thought maybe the RMS Layernorm epsilon was wrong - not 1e-5 but maybe 1e-6. For example [this](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct/blob/main/config.json) has `rms_norm_eps=1e-06`, whilst [this](https://huggingface.co/Qwen/Qwen2.5-32B/blob/main/config.json) has `rms_norm_eps=1e-05` . We also overrided it, but it did not work:

{% code overflow="wrap" %}

```bash
--override-kv qwen2.attention.layer_norm_rms_epsilon=float:0.000001 \
```

{% endcode %}

3. We also tested if tokenizer IDs matched between llama.cpp and normal Transformers courtesy of [@kalomaze](https://x.com/kalomaze/status/1897875332230779138). They matched, so this was not the culprit.

We provide our experimental results below:

{% file src="/files/Fb8C7pMzEAt2EMk6Y5Nv" %}
BF16 full precision with no sampling fix
{% endfile %}

{% file src="/files/Tza2MHehZX5GVoEgrsQj" %}
BF16 full precision with sampling fix
{% endfile %}

{% file src="/files/Q4ssXKESlEecT7UFam80" %}
Q4\_K\_M precision with no sampling fix
{% endfile %}

{% file src="/files/df2EQ852zfpf0oIdYi40" %}
Q4\_K\_M precision with sampling fix
{% endfile %}

## :pencil2: Tokenizer Bug Fixes

* We found a few issues as well specifically impacting finetuning! The EOS token is correct, but the PAD token should probably rather be `"<|vision_pad|>`" We updated it in: <https://huggingface.co/unsloth/QwQ-32B/blob/main/tokenizer_config.json>

```
"eos_token": "<|im_end|>",
"pad_token": "<|endoftext|>",
```

## :tools: Dynamic 4-bit Quants

We also uploaded dynamic 4bit quants which increase accuracy vs naive 4bit quantizations! We attach the QwQ quantization error plot analysis for both activation and weight quantization errors:

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fgit-blob-16157f53eff143c179be571a43f8b55000d94290%2FQwQ%20quantization%20errors.png?alt=media" alt=""><figcaption></figcaption></figure>

We uploaded dynamic 4-bit quants to: <https://huggingface.co/unsloth/QwQ-32B-unsloth-bnb-4bit>

Since vLLM 0.7.3 (2025 February 20th) <https://github.com/vllm-project/vllm/releases/tag/v0.7.3>, vLLM now supports loading Unsloth dynamic 4bit quants!

All our GGUFs are at <https://huggingface.co/unsloth/QwQ-32B-GGUF>!


# Cogito v2.1: How to Run Locally

Cogito v2.1 LLMs are one of the strongest open models in the world trained with IDA. Also v1 comes in 4 sizes: 70B, 109B, 405B and 671B, allowing you to select which size best matches your hardware.

{% hint style="success" %}
Deep Cogito v2.1 is an updated 671B MoE that is the most powerful open weights model as of 19 November 2025.
{% endhint %}

Cogito v2.1 comes in 1 671B MoE size, whilst Cogito v2 Preview is [Deep Cogito](https://www.deepcogito.com/)'s release of models spans 4 model sizes ranging from 70B to 671B. By using **IDA (Iterated Distillation & Amplification)**, these models are trained with the model internalizing the reasoning process using iterative policy improvement, rather than simply searching longer at inference time (like DeepSeek R1).

Deep Cogito is based in [San Fransisco, USA](https://techcrunch.com/2025/04/08/deep-cogito-emerges-from-stealth-with-hybrid-ai-reasoning-models/) (like Unsloth :flag\_us:) and we're excited to provide quantized dynamic models for all 4 model sizes! All uploads use Unsloth [Dynamic 2.0](/docs/basics/dynamic-3.0-ggufs) for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized these LLMs with minimal accuracy loss!

**Tutorials navigation:**

<a href="https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-671b-moe-in-llama.cpp" class="button secondary">Run 671B MoE</a><a href="https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-109b-moe-in-llama.cpp" class="button secondary">Run 109B MoE</a><a href="https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-405b-dense-in-llama.cpp" class="button secondary">Run 405B Dense</a><a href="https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/cogito-v2-how-to-run-locally#run-cogito-70b-dense-in-llama.cpp" class="button secondary">Run 70B Dense</a>

{% hint style="success" %}
Choose which model size fits your hardware! We upload 1.58bit to 16bit variants for all 4 model sizes!
{% endhint %}

## :gem: Model Sizes and Uploads

There are 4 model sizes:

1. 2 Dense models based off from Llama - 70B and 405B
2. 2 MoE models based off from Llama 4 Scout (109B) and DeepSeek R1 (671B)

<table data-full-width="false"><thead><tr><th>Model Sizes</th><th width="256.9999694824219">Recommended Quant &#x26; Link</th><th>Disk Size</th><th>Architecture</th></tr></thead><tbody><tr><td>70B Dense</td><td><a href="https://huggingface.co/unsloth/cogito-v2-preview-llama-70B-GGUF">UD-Q4_K_XL</a></td><td><strong>44GB</strong></td><td>Llama 3 70B</td></tr><tr><td>109B MoE</td><td><a href="https://huggingface.co/unsloth/cogito-v2-preview-llama-109B-MoE-GGUF">UD-Q3_K_XL</a></td><td><strong>50GB</strong></td><td>Llama 4 Scout</td></tr><tr><td>405B Dense</td><td><a href="https://huggingface.co/unsloth/cogito-v2-preview-llama-405B-GGUF">UD-Q2_K_XL</a></td><td><strong>152GB</strong></td><td>Llama 3 405B</td></tr><tr><td>671B MoE</td><td><a href="https://huggingface.co/unsloth/cogito-v2-preview-deepseek-671B-MoE-GGUF">UD-Q2_K_XL</a></td><td><strong>251GB</strong></td><td>DeepSeek R1</td></tr></tbody></table>

{% hint style="success" %}
Though not necessary, for the best performance, have your VRAM + RAM combined = to the size of the quant you're downloading. If you have less VRAM + RAM, then the quant will still function, just be much slower.
{% endhint %}

## 🐳 Run Cogito 671B MoE in llama.cpp

1. Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

{% code overflow="wrap" %}

```shellscript
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}

2. If you want to use `llama.cpp` directly to load models, you can do the below: (:IQ1\_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location.

{% hint style="success" %}
Please try out `-ot ".ffn_.*_exps.=CPU"` to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try `-ot ".ffn_(up|down)_exps.=CPU"` This offloads up and down projection MoE layers.

Try `-ot ".ffn_(up)_exps.=CPU"` if you have even more GPU memory. This offloads only up projection MoE layers.

And finally offload all layers via `-ot ".ffn_.*_exps.=CPU"` This uses the least VRAM.

You can also customize the regex, for example `-ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"` means to offload gate, up and down MoE layers but only from the 6th layer onwards.
{% endhint %}

```shellscript
export LLAMA_CACHE="unsloth/cogito-671b-v2.1-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/cogito-671b-v2.1-GGUF:UD-Q2_K_XL \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --top-p 0.95 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    --jinja \
    -ot ".ffn_.*_exps.=CPU"
```

3. Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-IQ1_S`(dynamic 1.78bit quant) or other quantized versions like `Q4_K_M` . We <mark style="background-color:green;">**recommend using our 2.7bit dynamic quant**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**`UD-Q2_K_XL`**</mark><mark style="background-color:green;">**&#x20;**</mark><mark style="background-color:green;">**to balance size and accuracy**</mark>. More versions at: <https://huggingface.co/unsloth/cogito-671b-v2.1-GGUF>

{% code overflow="wrap" %}

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0" # Can sometimes rate limit, so set to 0 to disable
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/cogito-671b-v2.1-GGUF",
    local_dir = "unsloth/cogito-671b-v2.1-GGUF",
    allow_patterns = ["*UD-IQ1_S*"], # Dynamic 1bit (168GB) Use "*UD-Q2_K_XL*" for Dynamic 2bit (251GB)
)
```

{% endcode %}

4. Edit `--threads 32` for the number of CPU threads, `--ctx-size 16384` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

## :mouse\_three\_button:Run Cogito 109B MoE in llama.cpp

1. Follow the same instructions as running the [671B model above](#run-cogito-671b-moe-in-llama.cpp).
2. Then run the below:

```shellscript
export LLAMA_CACHE="unsloth/cogito-v2-preview-llama-109B-MoE-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/cogito-v2-preview-llama-109B-MoE-GGUF:Q3_K_XL \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --min-p 0.01 \
    --top-p 0.9 \
    --ctx-size 16384 \
    --jinja \
    -ot ".ffn_.*_exps.=CPU"
```

## :deciduous\_tree:Run Cogito 405B Dense in llama.cpp

1. Follow the same instructions as running the [671B model above](#run-cogito-671b-moe-in-llama.cpp).
2. Then run the below:

```shellscript
export LLAMA_CACHE="unsloth/cogito-v2-preview-llama-405B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/cogito-v2-preview-llama-405B-GGUF:Q2_K_XL \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --min-p 0.01 \
    --top-p 0.9 \
    --jinja \
    --ctx-size 16384
```

## :sunglasses: Run Cogito 70B Dense in llama.cpp

1. Follow the same instructions as running the [671B model above](#run-cogito-671b-moe-in-llama.cpp).
2. Then run the below:

```shellscript
export LLAMA_CACHE="unsloth/cogito-v2-preview-llama-70B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/cogito-v2-preview-llama-70B-GGUF:Q4_K_XL \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --min-p 0.01 \
    --top-p 0.9 \
    --jinja \
    --ctx-size 16384
```

See <https://www.deepcogito.com/research/cogito-v2-1> for more details


# Gemma 4 - How to Run Locally

Run Google’s new Gemma 4 models locally, including E2B, E4B, 26B A4B, and 31B.

Gemma 4 is Google DeepMind’s new family of open models, including **12B**, **E2B**, **E4B**, **26B-A4B**, and **31B.** The multimodal, hybrid-thinking models support 140+ languages, up to **256K context**, and have dense and MoE variants. Gemma 4 is Apache-2.0 licensed and can run on your local device.

**Gemma-4-12B** is new and features unified text, image and audio support. It runs on **8GB** RAM (4-bit) or 14GB (8-bit). **Gemma-4-E2B** and **E4B** also support image and audio. Run on **5GB RAM** (4-bit) or 15GB (full 16-bit) via GGUF, MLX or NVFP4 quants.

<a href="/docs/models/gemma-4#run-gemma-4-tutorials" class="button primary">Run Gemma 4</a><a href="/docs/models/gemma-4/train" class="button secondary">Fine-tune Gemma 4</a><a href="/docs/models/gemma-4/qat" class="button primary">Gemma 4 QAT</a><a href="/docs/models/mtp#gemma-4-mtp" class="button secondary">Gemma 4 MTP</a>

{% hint style="success" %}
**NEW:** [**Gemma 4 MTP is here**](/docs/models/mtp)**! MTP enables 1.4-2.2x faster inference without accuracy loss. Run MTP directly in** [**Unsloth Studio**](/docs/models/mtp#unsloth-studio-mtp-guide)**.**
{% endhint %}

{% columns %}
{% column %}
**Gemma-4-26B-A4B** runs on **18GB** (4-bit) or 28GB (8-bit). **Gemma-4-31B** needs **20GB RAM** (4-bit) or 34GB (8-bit).

You can now run all GGUFs, [MLX](#mlx-dynamic-quants) and fine-tune Gemma 4 in [Unsloth Studio](#unsloth-studio-guide) (see right).

[**QAT** variants](/docs/models/gemma-4/qat) of Gemma 4 reduce memory requirements around 3x while preserving model quality.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FstfdTMsoBMmsbQsgQ1Ma%2Flandscape%20clip%20gemma.gif?alt=media&amp;token=eec5f2f7-b97a-4c1c-ad01-5a041c3e4013" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% hint style="success" %}
**Jun 9:** [Gemma 4 MTP](/docs/models/mtp) is here.

**Jun 5:** [Gemma 4 QAT](/docs/models/gemma-4/qat) is released.

**Jun 2:** Gemma 4 12B Unified is released.

**Apr 20:** We conducted [Gemma 4 GGUF Benchmarks](#unsloth-gguf-benchmarks) to help you pick the best quant.
{% endhint %}

### Usage Guide

Gemma 4 excels at reasoning, coding, tool use, long-context and agentic workflows, and multimodal tasks. The smaller E2B and E4B variants are designed for phones and laptops, while the larger models target medium-high CPU /VRAM systems such as PCs with NVIDIA RTX GPUs.

| Gemma 4 Variant | Details                                                          | Best fit                                              |
| --------------- | ---------------------------------------------------------------- | ----------------------------------------------------- |
| **E2B**         | <p>Dense + PLE (128K context)<br>Support: Text, Image, Audio</p> | For phone / edge inference, ASR, speech translation   |
| **E4B**         | <p>Dense + PLE (128K context)<br>Support: Text, Image, Audio</p> | Small model for laptops and fast local multimodal use |
| **12B Unified** | <p>Dense (256K context)<br>Support: Text, Image, Audio</p>       | Medium model for laptops and local multimodal use     |
| **26B-A4B**     | <p>MoE (256K context)<br>Support: Text, Image</p>                | Best speed / quality tradeoff for computer use        |
| **31B**         | <p>Dense (256K context)<br>Support: Text, Image</p>              | Strongest performance at slower inference             |

**See Gemma 4:** [**Performance benchmarks**](#official-gemma-benchmarks) **and** [**GGUF benchmarks**](#unsloth-gguf-benchmarks)**.**

**Should I pick 26B-A4B or 31B?**

* **26B-A4B** - balances speed and accuracy. Its MoE design makes it faster than 31B, with 4B active parameters. Pick it if RAM is limited and you are fine trading a bit of quality for speed.
* **31B** - currently the strongest Gemma 4 model. Pick it for maximum quality if you have enough memory and can accept slightly slower speeds.

### Hardware requirements

**Table: Gemma 4 Inference GGUF recommended hardware requirements** (units = total memory: RAM + VRAM, or unified memory). You can use Gemma 4 on MacOS, NVIDIA RTX GPUs etc.

| Gemma 4 variant |    4-bit |    8-bit | BF16 / FP16 |
| --------------- | -------: | -------: | ----------: |
| **E2B**         |     4 GB |   5–8 GB |       10 GB |
| **E4B**         | 5.5–6 GB |  9–12 GB |       16 GB |
| **12B Unified** |   7–8 GB | 13–14 GB |       25 GB |
| **26B A4B**     | 16–18 GB | 28–30 GB |       52 GB |
| **31B**         | 17–20 GB | 34–38 GB |       62 GB |

{% hint style="info" %}
As a rule of thumb, your total available memory should at least exceed the size of the quantized model you download. If it does not, llama.cpp can still run using partial RAM / disk offload, but generation will be slower. You will also need more compute, depending on the context window you use.
{% endhint %}

### Recommended Settings

It is recommended to use Google's default Gemma 4 parameters:

* `temperature = 1.0`
* `top_p = 0.95`
* `top_k = 64`

{% hint style="info" %}
Gemma 4's max context is **128K** for **E2B** / **E4B** and `262,144` for **12B** / **26B A4B** / **31B**.
{% endhint %}

#### Thinking Mode

Compared to older Gemma chat templates, Gemma 4 uses the standard **`system`**, **`assistant`**, and **`user`** roles and adds explicit thinking control.

**How to enable thinking:**

Add the token **`<|think|>`** at the **start of the system prompt**.

{% columns %}
{% column %}
**Thinking enabled**

```
<|think|>
You are a careful coding assistant. Explain your answer clearly.
```

{% endcolumn %}

{% column %}
**Thinking disabled**

```
You are a careful coding assistant. Explain your answer clearly.
```

{% endcolumn %}
{% endcolumns %}

**Output behavior:**

{% columns %}
{% column %}
When thinking is enabled, the model outputs its internal reasoning channel before the final answer.

```
<|channel>thought
[internal reasoning]
<channel|>
[final answer]
```

{% endcolumn %}

{% column %}
When thinking is disabled, the larger models may still emit an **empty thought block** before the final answer.

```
<|channel>thought
<channel|>
[final answer]
```

{% endcolumn %}
{% endcolumns %}

**For example using "**&#x57;hat is the capital of France?":

{% code overflow="wrap" %}

```
<bos><|turn>system\n<|think|><turn|>\n<|turn>user\nWhat is the capital of France?<turn|>\n<|turn>model\n
```

{% endcode %}

**then it outputs with:**

{% code overflow="wrap" %}

```
<|channel>thought\nThe user is asking for the capital of France.\nThe capital of France is Paris.<channel|>The capital of France is Paris.<turn|>
```

{% endcode %}

**Multi-turn chat rule:**

For multi-turn conversations, **only keep the final visible answer in chat history**. Do **not** feed prior thought blocks back into the next turn.

{% code overflow="wrap" %}

```
<bos><|turn>user\nWhat is 1+1?<turn|>\n<|turn>model\n2<turn|>\n<|turn>user\nWhat is 1+1?<turn|>\n<|turn>model\n2<turn|>\n<|turn>user\nWhat is 1+1?<turn|>\n<|turn>model\n2<turn|>\n<|turn>user\nWhat is 1+1?<turn|>\n<|turn>model\n2<turn|>\n
```

{% endcode %}

**How to disable thinking:**

Note `llama-cli` might not work reliably, so use `llama-server` for disabling reasoning:

{% hint style="warning" %}
To [disable thinking / reasoning](#how-to-enable-or-disable-reasoning-and-thinking), use `--chat-template-kwargs '{"enable_thinking":false}'`

If you're on **Windows** Powershell, use: `--chat-template-kwargs "{\"enable_thinking\":false}"`

Use 'true' and 'false' interchangeably.
{% endhint %}

## Run Gemma 4 Tutorials

Because Gemma 4 GGUFs comes in several sizes, the recommended starting point for the small models is 8-bit and the larger models is [**Dynamic**](/docs/basics/dynamic-3.0-ggufs) **4-bit**. [Gemma 4 GGUFs](https://huggingface.co/collections/unsloth/gemma-4) or [MLX](#mlx-dynamic-quants) or [NVFP4](#nvfp4-guide):

| [E2B](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF) | [E4B](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | [12b](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF) | [26B-A4B](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF) | [31B](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) |
| --------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------- |

<a href="/docs/models/gemma-4#unsloth-studio-guide" class="button primary">🦥 Unsloth Desktop Guide</a><a href="/pages/VnmWq1kNppQrTqCI6aLH#llama.cpp-guide" class="button primary">🦙 Llama.cpp Guide</a><a href="/docs/models/gemma-4#nvfp4-guide" class="button primary">NVFP4 Guide</a>

{% columns %}
{% column %}
**You can run and train Gemma 4 for free with a UI in our** [**Unsloth Studio**](/docs/new/studio)✨ **notebook:**
{% endcolumn %}

{% column %}
{% embed url="<https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

### 🦥 Unsloth Guide

Gemma 4 can now be run and fine-tuned in [Unsloth Desktop](/docs/new/studio), our new open-source desktop UI for local AI. Unsloth Studio lets you run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio#run-models-locally) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpAGvwjGD0iVMZKBoyu7m%2Fgreeennn.png?alt=media&amp;token=d17a5528-8375-444c-9aff-f9e9f7903bcd" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows PowerShell:**

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Launch Unsloth

**MacOS, Linux, WSL and Windows:**

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

Then open `http://127.0.0.1:8888` or your specific URL in your browser.

**Launch Unsloth securely with HTTPS and Cloudflare**

**NEW!** Unsloth now provides a secure way to launch Unsloth over HTTPS through a free Cloudflare tunnel. Use the below (works in Windows, Mac & Linux):

```bash
unsloth studio --secure
```

{% endstep %}

{% step %}

#### Search and download Gemma 4

On first launch you will need to create a password to secure your account and sign in again.

Then go to the [Unsloth Chat](/docs/new/studio/chat) tab and search for Gemma 4 in the search bar and download your desired model and quant. Unsloth supports the latest Gemma-4-12B Unified model.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpYoNILI8NFMl8QaQlc7V%2FScreenshot%202026-04-02%20at%2010.37.32%E2%80%AFPM.png?alt=media&amp;token=18d5918e-4f71-4e0e-b8c9-464097389835" alt="" width="375"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

#### Run Gemma 4

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings. You can run GGUFs and MLX files.

For more information, you can view our [Unsloth Studio inference guide](/docs/new/studio/chat).

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVrLgXwplAMcvkU4owjPk%2F26b%20gif.gif?alt=media&amp;token=8a569952-c152-435f-b815-c9f295619587" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}
{% endstepper %}

### 🦙 Llama.cpp Guide

For this guide we will be utilizing Dynamic 4-bit for the 12B, 26B-A4B and 31B, and 8-bit for E2B and E4B.  See: [Gemma 4 GGUF collection](https://huggingface.co/collections/unsloth/gemma-4)

For these tutorials, we will using [llama.cpp](llama.cpphttps://github.com/ggml-org/llama.cpp) for fast local inference, especially if you have a CPU.

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` **on** [**GitHub here**](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can follow commands below, according to each model. `UD-Q4_K_XL` is the quantization type. You can also download via Hugging Face (step 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. There is no need to set context length as llama.cpp automatically uses the exact amount required.

{% hint style="warning" %}
To [disable thinking / reasoning](#how-to-enable-or-disable-reasoning-and-thinking), use: `--chat-template-kwargs '{"enable_thinking":false}'`

**Windows** Powershell: `--chat-template-kwargs "{\"enable_thinking\":false}"`

Use '`true`' and '`false`' interchangeably.
{% endhint %}

**12B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-12B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

**26B-A4B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-26B-A4B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

**31B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-31B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

**E4B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-E4B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

**E2B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-E2B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E2B-it-GGUF:Q8_0 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

{% endstep %}

{% step %}
You can also download the model manually as well via the code below (after installing `pip install huggingface_hub`). You can choose `UD-Q4_K_XL` or other quantized versions like `Q8_0` . If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

```bash
hf download unsloth/gemma-4-26B-A4B-it-GGUF \
    --local-dir unsloth/gemma-4-26B-A4B-it-GGUF \
    --include "*mmproj-BF16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

{% endstep %}

{% step %}
Then run the model in conversation mode (with vision `mmproj-F16`):

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

{% endcode %}
{% endstep %}

{% step %}

#### Llama-server deployment

To deploy Gemma-4 on llama-server, use:

```bash
./llama.cpp/llama-server \
    --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64 \
    --alias "unsloth/gemma-4-26B-A4B-it-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

{% endstep %}
{% endstepper %}

### MLX Dynamic Quants

We also uploaded dynamic 4bit and 8bit quants as a first trial for MacOS device! The MLX quants support **vision.**

{% hint style="success" %}
All MLX quants now work in[ Unsloth Studio](#unsloth-studio-guide)!
{% endhint %}

| Gemma 4 | 4-bit MLX                                                             | 8-bit MLX                                                          |
| ------- | --------------------------------------------------------------------- | ------------------------------------------------------------------ |
| 31B     | [link](https://huggingface.co/unsloth/gemma-4-31b-it-UD-MLX-4bit)     | [link](https://huggingface.co/unsloth/gemma-4-31b-it-MLX-8bit)     |
| 26B-A4B | [link](https://huggingface.co/unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit) | [link](https://huggingface.co/unsloth/gemma-4-26b-a4b-it-MLX-8bit) |
| E4B     | [link](https://huggingface.co/unsloth/gemma-4-E4B-it-UD-MLX-4bit)     | [link](https://huggingface.co/unsloth/gemma-4-E4B-it-MLX-8bit)     |
| E2B     | [link](https://huggingface.co/unsloth/gemma-4-E2B-it-UD-MLX-4bit)     | [link](https://huggingface.co/unsloth/gemma-4-E2B-it-MLX-8bit)     |

To try them out use:

{% code overflow="wrap" %}

```bash
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/scripts/install_gemma4_mlx.sh | sh
source ~/.unsloth/unsloth_gemma4_mlx/bin/activate
python -m mlx_vlm.chat --model unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit
```

{% endcode %}

### **NVFP4 Guide**

We uploaded [Dynamic NVFP4](/docs/basics/nvfp4) Gemma 4 quants for faster 4-bit inference on NVIDIA Blackwell GPUs. are the hardware requirements for models which you can use including Gemma 4. Also see the overall speed boost you will achieve:

| Gemma 4 variant                                                    | Required VRAM | Faster than BF16 |
| ------------------------------------------------------------------ | ------------: | ---------------: |
| [E2B](https://huggingface.co/unsloth/gemma-4-E2B-it-NVFP4)         |          7 GB |     1.12× faster |
| [E4B](https://huggingface.co/unsloth/gemma-4-E4B-it-NVFP4)         |          9 GB |     1.22× faster |
| [12B Unified](https://huggingface.co/unsloth/gemma-4-12b-it-NVFP4) |         11 GB |     1.26× faster |
| [26B A4B](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-NVFP4) |         26 GB |     1.41× faster |
| [31B](https://huggingface.co/unsloth/gemma-4-31B-it-NVFP4)         |         32 GB |     1.45× faster |

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZw91MhTauyOZGP5HT7bl%2Fimage.png?alt=media&amp;token=ae0051a9-af39-4ef7-8ed2-9f8c1fff33fc" alt="" width="563"><figcaption></figcaption></figure>

#### **vLLM Tutorial:**

To run NVFP4 quants, see below for commands to run Gemma-4-26B-A4B in [vLLM](/docs/basics/inference-and-deployment/vllm-guide) and [SGLang](/docs/basics/inference-and-deployment/sglang-guide) (you can change model name to `gemma-4-31B-it-NVFP4` etc.) Also do NOT select any MoE backend - leave vLLM to select it - for eg Marlin is 2.5x slower! See [#marlin-vs-flashinfer-vs-cutlass-vs-cute-dsl](#marlin-vs-flashinfer-vs-cutlass-vs-cute-dsl "mention")If you have a DGX Spark, see [#dgx-spark-serving](#dgx-spark-serving "mention") you must use `--moe-backend flashinfer_b12x` or you will get much slower inference.

To install vLLM in a separate venv:

{% code overflow="wrap" expandable="true" %}

```bash
uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto
```

{% endcode %}

Then to serve the 26B MoE variant:

```shell
vllm serve unsloth/gemma-4-26B-A4B-it-NVFP4
```

Change `unsloth/gemma-4-26B-A4B-it-NVFP4` to any [available NVFP4](/docs/basics/nvfp4#overview) quant name!

To enable MTP / speculative decoding (faster decode but somewhat less throughput), use:

```bash
vllm serve unsloth/gemma-4-26B-A4B-it-NVFP4
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
```

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

{% code overflow="wrap" expandable="true" %}

```bash
sudo apt-get update
sudo apt-get install -y ffmpeg
```

{% endcode %}

#### **DGX Spark with NVFP4 quants**

To ensure DGX Spark has the correct kernels (or you will get **2x SLOWER inference**), first check:

{% code overflow="wrap" expandable="true" %}

```bash
python -c "
import torch; from vllm.utils.flashinfer import has_flashinfer_b12x_gemm as g, has_flashinfer_b12x_moe as m
cap = torch.cuda.get_device_capability(); print('cap', cap, '| b12x gemm', g(), '| b12x moe', m()); assert cap[0] == 12 and g() and m(), 'b12x unavailable: serving would degrade to marlin W4A16'"
```

{% endcode %}

which should NOT error out - if it did, please update vllm or reinstall via:

{% code overflow="wrap" expandable="true" %}

```bash
uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto
```

{% endcode %}

Then to serve in vLLM for DGX Spark:

{% code overflow="wrap" expandable="true" %}

```shellscript
export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/gemma-4-26B-A4B-it-NVFP4 --moe-backend flashinfer_b12x
```

{% endcode %}

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

{% code overflow="wrap" expandable="true" %}

```bash
sudo apt-get update
sudo apt-get install -y ffmpeg
```

{% endcode %}

#### **SGLang Tutorial:**

```bash
python -m sglang.launch_server --model-path unsloth/gemma-4-31B-it-NVFP4 --speculative-algorithm NEXTN \
     --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
```

### Ollama Guide

Ollama now supports Unsloth GGUFs well now. Use `curl -fsSL https://ollama.com/install.sh | sh` to install Ollama on Linux or `irm https://ollama.com/install.ps1 | iex` for Windows.\
\
To use a single quant file (under 50GB) use:

{% code overflow="wrap" %}

```bash
ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL
```

{% endcode %}

For multiple shards like larger BF16 shards do:

{% code overflow="wrap" %}

```bash
pip install -U huggingface_hub

# Download mmproj and BF16 in 2 calls
hf download unsloth/gemma-4-26B-A4B-it-GGUF --include "BF16/*" \
    --local-dir gemma4
hf download unsloth/gemma-4-26B-A4B-it-GGUF --include "mmproj-BF16.gguf" \
    --local-dir gemma4

mv gemma4/mmproj-BF16.gguf gemma4/BF16/
echo "FROM ./gemma4/BF16" > Modelfile

ollama create unsloth-gemma4 -f Modelfile
ollama run unsloth-gemma4
```

{% endcode %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FjflAyMZg9ZaSmeUL7LlQ%2Fimage.png?alt=media&amp;token=ddda55e7-0526-4fe0-99bc-d0cb4d05f7fa" alt="" width="563"><figcaption></figcaption></figure></div>

{% hint style="info" %}
If you see `Error: 500 Internal Server Error: unable to load model` update Ollama via `curl -fsSL https://ollama.com/install.sh | sh` or use the Powershell one.
{% endhint %}

## Gemma 4 Best Practices

### Prompting examples

#### Simple reasoning prompt

```
System:
<|think|>
You are a precise reasoning assistant.

User:
A train leaves at 8:15 AM and arrives at 11:47 AM. How long was the journey?
```

#### OCR / document prompt

For OCR, use a **high visual token budget** like **560** or **1120**.

```
[image first]
Extract all text from this receipt. Return line items, total, merchant, and date as JSON.
```

#### Multi-modal comparison prompt

```
[image 1]
[image 2]
Compare these two screenshots and tell me which one is more likely to confuse a new user.
```

#### Audio ASR prompt

```
Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.

Follow these specific instructions for formatting the answer:
* Only output the transcription, with no newlines.
* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
```

#### Audio translation prompt

```
Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}. When formatting the answer, first output the transcription in {SOURCE_LANGUAGE}, then one newline, then output the string '{TARGET_LANGUAGE}: ', then the translation in {TARGET_LANGUAGE}.
```

### Multi-modal Settings

For best results with multimodal prompts, put multimodal content first:

* Put **image and/or audio before text**.
* For video, pass a sequence of frames first, then the instruction.

#### Audio and video limits

* **Audio** is available on **12B**, **E2B** and **E4B** only.
* Audio supports a maximum of **30 seconds**.
* Video supports a maximum of **60 seconds** assuming **1 frame per second** processing.

#### Audio prompt templates

**ASR prompt**

```
Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.

Follow these specific instructions for formatting the answer:
* Only output the transcription, with no newlines.
* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
```

**Speech translation prompt**

```
Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}.
When formatting the answer, first output the transcription in {SOURCE_LANGUAGE}, then one newline, then output the string '{TARGET_LANGUAGE}: ', then the translation in {TARGET_LANGUAGE}.
```

## 📊 Benchmarks

### Unsloth GGUF Benchmarks

We conducted Mean KL Divergence benchmarks for Gemma 4 GGUFs across providers to help you pick the best quant (lower is better).

* KL Divergence puts all Unsloth GGUFs on the SOTA Pareto frontier
* KLD shows how well a quantized model matches the original BF16 output distribution, indicating retained accuracy.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FtRVN97QDzO0Pq7SscC7x%2Fgemma%20426b%20bench.png?alt=media&amp;token=80b4da76-efe9-4554-8e31-cca6494d456c" alt=""><figcaption><p>26B A4B - KLD benchmarks (lower is better)</p></figcaption></figure></div>

### Official Gemma Benchmarks

**Text/Code Benchmarks**

| Benchmark           | Gemma 4 31B | Gemma 4 26B A4B | Gemma 4 12B Unified | Gemma 4 E4B | Gemma 4 E2B | Gemma 3 27B (no think) |
| ------------------- | ----------- | --------------- | ------------------- | ----------- | ----------- | ---------------------- |
| MMLU Pro            | 85.2%       | 82.6%           | 77.2%               | 69.4%       | 60.0%       | 67.6%                  |
| AIME 2026 no tools  | 89.2%       | 88.3%           | 77.5%               | 42.5%       | 37.5%       | 20.8%                  |
| LiveCodeBench v6    | 80.0%       | 77.1%           | 72.0%               | 52.0%       | 44.0%       | 29.1%                  |
| Codeforces ELO      | 2150        | 1718            | 1659                | 940         | 633         | 110                    |
| GPQA Diamond        | 84.3%       | 82.3%           | 78.8%               | 58.6%       | 43.4%       | 42.4%                  |
| Tau2                | 76.9%       | 68.2%           | 69.0%               | 42.2%       | 24.5%       | 16.2%                  |
| HLE no tools        | 19.5%       | 8.7%            | 5.2%                | -           | -           | -                      |
| HLE with search     | 26.5%       | 17.2%           | -                   | -           | -           | -                      |
| BigBench Extra Hard | 74.4%       | 64.8%           | 53.0%               | 33.1%       | 21.9%       | 19.3%                  |
| MMMLU               | 88.4%       | 86.3%           | 83.4%               | 76.6%       | 67.4%       | 70.7%                  |

**Vision Benchmarks**

| MMMU Pro                           | 76.9% | 73.8% | 69.1% | 52.6% | 44.2% | 49.7% |
| ---------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- |
| OmniDocBench 1.5 (lower is better) | 0.131 | 0.149 | 0.164 | 0.181 | 0.290 | 0.365 |
| MATH-Vision                        | 85.6% | 82.4% | 79.7% | 59.5% | 52.4% | 46.0% |
| MedXPertQA MM                      | 61.3% | 58.1% | 48.7% | 28.7% | 23.5% | -     |

**Audio Benchmarks**

| CoVoST                          | -     | -     | 38.5<sup>\*</sup>  | 35.54 | 33.47 | -     |
| ------------------------------- | ----- | ----- | ------------------ | ----- | ----- | ----- |
| FLEURS (lower is better)        | -     | -     | 0.069<sup>\*</sup> | 0.08  | 0.09  | -     |
| **Long Context**                |       |       |                    |       |       |       |
| MRCR v2 8 needle 128k (average) | 66.4% | 44.1% | 43.4%              | 25.4% | 19.1% | 13.5% |

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FfKaFMy7LHQYNKpfsf7Zy%2Fgemma%204%20banner.png?alt=media&amp;token=8bd8d0e0-ccb6-4ded-b99b-2c8a18370ae5" alt=""><figcaption></figcaption></figure></div>


# Gemma 4 QAT

Run Google Gemma 4 QAT models locally, including E2B, E4B, 12B, 26B-A4B, and 31B.

Gemma 4 QAT (Quantization-Aware Training) is Google DeepMind’s new [Gemma 4](/docs/models/gemma-4) variants designed to **reduce memory requirements while preserving model quality**. This makes it possible to run larger models, such as **Gemma 4 26B-A4B**, locally on consumer GPUs with as little as **16GB of RAM**.

Gemma 4 QAT is trained with quantization in mind, allowing 4-bit format to have \~**72% lower memory usage** with **near original performance**. 2 special mobile quants of E2B and E4B are also provided which uses a mixture of quant widths.

Converting to `Q4_0` from QAT naively gets only 70.2% top-1 % accuracy for 26B-A4B. [We applied our Unsloth Dynamic method](#qat-analysis) to push it up to **85.6% (+15.6%) whilst also being** [**200MB smaller**](#usage-guide)!

Gemma 4 QAT includes: **E2B**, **E4B**, **12B, 26B-A4B**, and **31B.** They are multimodal, hybrid-thinking models that support 140+ languages and up to **256K context.**

{% columns %}
{% column %} <a href="/docs/models/gemma-4/qat#run-gemma-4-qat-tutorials" class="button primary">Run Gemma 4 QAT</a><a href="/docs/models/gemma-4/qat#qat-analysis" class="button secondary">QAT Analysis</a>

**Gemma-4-E2B** QAT runs on 3GB RAM, **E4B** on 5G&#x42;**, 12B** on 7G&#x42;**, 26-A4B** on 15GB and **31B** on 18G&#x42;**.**

We name our Gemma 4 QAT GGUFs as `UD-Q4_K_XL` as we found q4\_0 to degrade accuracy despite being bigger. See our [Gemma 4 QAT GGUFs](https://huggingface.co/collections/unsloth/gemma-4-qat).

To compare `int4` quantization, see the original vs. QAT size differences below. QAT uses \~72% less memory whilst retaining nearly all its original accuracy:
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FGLHNSIo3qe4ByThVZDzm%2Fgemma%204%20qat.png?alt=media&amp;token=ae72b8e1-def7-447c-bafc-67514354df7d" alt="" width="563"><figcaption><p>Visualization of how Gemma 4 Mobile QAT works.</p></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

| Gemma 4     | QAT (int4) GGUF | Original BF16 | Percentage change |
| ----------- | --------------: | ------------: | ----------------: |
| **E2B**     |         2.62 GB |       9.31 GB |            71.86% |
| **E4B**     |         4.22 GB |       15.1 GB |            72.05% |
| **12B**     |         6.72 GB |       23.8 GB |            71.76% |
| **26B A4B** |         14.2 GB |       50.5 GB |            71.88% |
| **31B**     |         17.3 GB |       61.4 GB |            71.82% |

### Usage Guide

Gemma 4 QAT variants for E2B and E4B are designed for phones and laptops, while the larger 26B-A4B and 31B QAT models now work on laptops rather than just strong home GPUs.

There is **only one GGUF file** for each Gemma 4 model because we found that precisions higher than the uploaded `UD-Q4_K_XL` version degrade accuracy rather than improve it. Use the original non QAT Q4\_0 quants [here](https://huggingface.co/collections/unsloth/gemma-4).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FrTygbMBlZcUER9GUMzKg%2Fimage.png?alt=media&amp;token=85c3c44b-0f42-477a-9ce2-30b085e21b5c" alt="" width="563"><figcaption></figcaption></figure>

### Hardware requirements

**Table: Gemma 4 QAT Inference GGUF recommended hardware requirements** (units = total memory: RAM + VRAM, or unified memory).

| Gemma 4 QAT     | Requirements |
| --------------- | -----------: |
| **E2B** QAT     |         3 GB |
| **E4B** QAT     |         5 GB |
| **12B** QAT     |         7 GB |
| **26B A4B** QAT |        15 GB |
| **31B** QAT     |        18 GB |

{% hint style="info" %}
The table assumes \~8K tokens of context. llama.cpp’s default 256K context for 26B A4B adds \~5 GB of KV cache. On memory-limited CPU-only machines, use `-c 8192` or `-c 16384`; `--fit` currently ignores CPU repacked weights.
{% endhint %}

### Recommended Settings

The QAT checkpoints use the same recommended Gemma 4 settings:

* `temperature = 1.0`
* `top_p = 0.95`
* `top_k = 64`

{% hint style="info" %}
Gemma 4's max context is **128K** for **E2B**, **E4B** and **256K** for **12B**, **26B A4B**, **31B**.
{% endhint %}

## QAT Analysis

We found that naively converting the QAT Q4\_0 checkpoint to Q4\_0 in llama.cpp land actually degraded accuracy and was not actually aligned with the BF16 QAT lattice for Q4\_0. We applied our Unsloth dynamic method to force a better agreement between the llama.cpp compatible Q4\_0 format and the true BF16 QAT Q4\_0 format, and managed to both make the quants smaller (Q6\_K wasn't needed for embeddings), and also more accurate!

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FcBHRs9twWiAnjgtm4EKf%2Fimage.png?alt=media&amp;token=023d7771-dba1-4a1e-a8c4-da6ac98921f7" alt=""><figcaption></figcaption></figure>

Below is a table of KLD and Top 1% accuracy and Disk space. You can see our versions dramatically improve on 99.9% KLD and mean KLD. **E2B for example has a mean KLD of 0.00173 vs 0.05109 (29x better relatively) for a naive Q4\_0 quantization, and ours is even 22% smaller!**

The main issue is converting from QAT BF16 to llama.cpp's Q4\_0 format is not lossless. llama.cpp uses F16 scales, whilst QAT BF16 uses BF16 scales, and the scales are not determined optimally in llama.cpp land.

Naive conversion gets 24.77% byte exactness to BF16 QAT, whilst we found we can push it to 99.96% using some hacks!

<table><thead><tr><th width="100">Model</th><th>Method</th><th>Disk (GB)</th><th>99.9% KLD</th><th>Mean KLD</th><th width="77.5999755859375">Top-1 %</th></tr></thead><tbody><tr><td>E2B</td><td>Unsloth</td><td><strong>2.62</strong></td><td>0.0557</td><td>0.00173</td><td><strong>98.16</strong></td></tr><tr><td>E2B</td><td>Q4_0</td><td>3.35</td><td>1.0513</td><td>0.05109</td><td>89.29</td></tr><tr><td>E4B</td><td>Unsloth</td><td><strong>4.22</strong></td><td>0.0536</td><td>0.00121</td><td><strong>98.54</strong></td></tr><tr><td>E4B</td><td>Q4_0</td><td>5.15</td><td>0.6722</td><td>0.03778</td><td>90.94</td></tr><tr><td>26B</td><td>Unsloth</td><td><strong>14.25</strong></td><td>2.7087</td><td>0.09788</td><td><strong>85.63</strong></td></tr><tr><td>26B</td><td>Q4_0</td><td>14.44</td><td>4.5420</td><td>0.36094</td><td>70.20</td></tr><tr><td>31B</td><td>Unsloth</td><td><strong>17.29</strong></td><td>1.3659</td><td>0.01403</td><td><strong>96.67</strong></td></tr><tr><td>31B</td><td>Q4_0</td><td>17.65</td><td>3.0030</td><td>0.09349</td><td>87.91</td></tr><tr><td>12B</td><td>Unsloth</td><td><strong>6.72</strong></td><td>9.2740</td><td>0.13288</td><td><strong>88.76</strong></td></tr><tr><td>12B</td><td>Q4_0</td><td>6.98</td><td>14.7323</td><td>0.50702</td><td>74.08</td></tr></tbody></table>

## Mobile Mixture QAT

The Gemma-4 team also released special mobile mixture QAT versions of Gemma-4-E2B-it and Gemma-4-E4B-it. We also faithfully converted them to llama.cpp compatible format, and also recovered nearly all accuracy as well. We used TQ2\_0 for the 2-bit layers and did a negative scaler.

We made UD-Q2\_K\_XL quants for both E2B and E4B.

|                        | E2B mobile          | E4B mobile          |
| ---------------------- | ------------------- | ------------------- |
| Size                   | 2.19 GB             | 3.22 GB             |
| 2-bit (TQ2\_0) tensors | 61 (incl. deep MLP) | 2 (embeddings only) |
| Mean KLD vs BF16       | 0.00409             | 0.00102             |
| Top-1 %                | 97.82%              | 98.76%              |
| Base PPL               | \~103               | 42.4                |

See [gemma-4-E2B-it-qat-GGUF](https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF) and [gemma-4-E4B-it-qat-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF) for `UD-Q2_K_XL`.

## Run Gemma 4 QAT Tutorials

Because Gemma 4 GGUFs comes in several sizes, the recommended starting point for the small models is 8-bit and the larger models is **Dynamic 4-bit**. [Gemma 4 GGUFs](https://huggingface.co/collections/unsloth/gemma-4-qat):

| [E2B](https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF) | [E4B](https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF) | [12b](https://huggingface.co/unsloth/gemma-4-12b-it-qat-GGUF) | [26B-A4B](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF) | [31B](https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF) |
| ------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------- |

<a href="/docs/models/gemma-4/qat#unsloth-studio-guide" class="button primary">🦥 Unsloth Studio Guide</a><a href="/pages/9kjF1F7Gsb0dgOA9lcpl#llama.cpp-guide" class="button primary">🦙 Llama.cpp Guide</a>

{% columns %}
{% column %}
**You can run and train Gemma 4 QAT for free with a UI in our** [**Unsloth Studio**](/docs/new/studio)✨ **notebook:**
{% endcolumn %}

{% column %}
{% embed url="<https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

### 🦥 Unsloth Studio Guide

Gemma 4 QAT can now be run and trained in [Unsloth Studio](/docs/new/studio), our new open-source web UI for local AI. Unsloth Studio lets you run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio#run-models-locally) (Python, Bash)
* [Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via llama.cpp
* [Train LLMs](/docs/new/studio#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVrLgXwplAMcvkU4owjPk%2F26b%20gif.gif?alt=media&amp;token=8a569952-c152-435f-b815-c9f295619587" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

Run in your terminal:

**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows PowerShell:**

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Launch Unsloth

**MacOS, Linux, WSL and Windows:**

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

Then open `http://127.0.0.1:8888` (or your specific URL) in your browser.
{% endstep %}

{% step %}

#### Search and download Gemma 4 QAT

On first launch you will need to create a password to secure your account and sign in again.

Then go to the [Unsloth Chat](/docs/new/studio/chat) tab and search for Gemma 4 in the search bar and download your desired model and quant.
{% endstep %}

{% step %}

#### Run Gemma 4 QAT

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth Studio inference guide](/docs/new/studio/chat).

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FVrLgXwplAMcvkU4owjPk%2F26b%20gif.gif?alt=media&amp;token=8a569952-c152-435f-b815-c9f295619587" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}
{% endstepper %}

### 🦙 Llama.cpp Guide

For this guide there is no need to select quantization type since there is only one: `UD-Q4_K_XL`.  See: [Gemma 4 QAT collection](https://huggingface.co/collections/unsloth/gemma-4-qat). For these tutorials, we will using [llama.cpp](llama.cpphttps://github.com/ggml-org/llama.cpp) for fast local inference, especially if you have a CPU.

{% stepper %}
{% step %}
Obtain the latest `llama.cpp` **on** [**GitHub here**](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you want to use `llama.cpp` directly to load models, you can follow commands below, according to each model. `UD-Q4_K_XL` is the ONLY quantization type. You can also download via Hugging Face (step 3). This is similar to `ollama run` . Use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location.

**26B-A4B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-26B-A4B-it-qat-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

**31B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-31B-it-qat-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-31B-it-qat-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

**E4B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-E4B-it-qat-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

**E2B:**

```bash
export LLAMA_CACHE="unsloth/gemma-4-E2B-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E2B-it-qat-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub hf_transfer` ). You can choose `UD-Q4_K_XL` or other quantized versions like `Q8_0` . If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

```bash
hf download unsloth/gemma-4-26B-A4B-it-qat-GGUF \
    --local-dir unsloth/gemma-4-26B-A4B-it-qat-GGUF \
    --include "*mmproj-BF16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

{% endstep %}

{% step %}
Then run the model in conversation mode (with vision `mmproj-F16`):

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-26B-A4B-it-qat-GGUF/mmproj-BF16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64
```

{% endcode %}
{% endstep %}

{% step %}

### Llama-server deployment

To deploy Gemma-4 on llama-server, use:

```bash
./llama.cpp/llama-server \
    --model unsloth/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-26B-A4B-it-qat-GGUF/mmproj-BF16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64 \
    --alias "unsloth/gemma-4-26B-A4B-it-qat-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

{% hint style="warning" %}
To [disable thinking / reasoning](#how-to-enable-or-disable-reasoning-and-thinking), use `--chat-template-kwargs '{"enable_thinking":false}'`

If you're on **Windows** Powershell, use: `--chat-template-kwargs "{\"enable_thinking\":false}"`

Use 'true' and 'false' interchangeably.
{% endhint %}
{% endstep %}
{% endstepper %}


# Gemma 4 Fine-tuning Guide

Train Gemma 4 by Google with Unsloth.

You can now train Google's [Gemma 4](https://unsloth.ai/docs/models/gemma-4) 12B, E2B, E4B, 26B-A4B and 31B with [**Unsloth**](https://github.com/unslothai/unsloth). Unsloth supports all vision, text, audio and RL fine-tuning for Gemma 4.

* Unsloth trains Gemma 4 **\~1.5x faster** with **\~60% less VRAM** than FA2 setups (no accuracy loss)
* We fixed many universal [bugs for Gemma 4 training](#bug-fixes--tips) (not derived from Unsloth).
* Gemma 4 E2B trains on **8GB VRAM**. E4B requires 10GB VRAM.

<a href="/docs/models/gemma-4/train#quickstart" class="button primary" data-icon="bolt">Quickstart</a><a href="/docs/models/gemma-4/train#bug-fixes--tips" class="button secondary" data-icon="sparkle">Bug Fixes + Tips</a>

Fine-tune Gemma 4 via our **free** **Google Colab notebooks**:

| [ (Unsloth)](https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb) | [**31B** (Kaggle)](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth) | [E4B **(Vision + Text)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E4B\)-Vision.ipynb) | [E4B **(Audio)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E4B\)-Audio.ipynb) | [E2B **(RL GRPO)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)_Reinforcement_Learning_Sudoku_Game.ipynb) |
| -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |

{% columns %}
{% column %}
You can run and train Gemma 4 for free with a UI in our [Unsloth Studio](/docs/new/studio)✨ notebook:

You can view more [notebooks here](#unsloth-core-code-based-guide).
{% endcolumn %}

{% column %}
{% embed url="<https://colab.research.google.com/github/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb>" %}
{% endcolumn %}
{% endcolumns %}

* You can also train Gemma 4 with [reinforcement learning](#reinforcement-learning-rl) (RL) on 9GB VRAM.
* Gemma 4 E2B LoRA works on 8-10GB VRAM. E4B LoRA requires 17GB VRAM.
* **31B QLoRA works with 22GB** and 26B-A4B LoRA needs >40GB
* **Exporting**/saving models to GGUF etc. and full fine-tuning **(FFT)** works as well.

### :bug: Bug fixes + Tips

{% hint style="success" %}
If you see **Gemma-4 E2B and E4B having a loss of 13-15, this is perfectly normal** - this is a common quirk of multimodal models. This also happened on Gemma-3N, Llama Vision, Mistral vision models and more.

**Gemma 26B and 31B have lower loss at 1-3 or lower. Vision will be 2x higher so 3-5**
{% endhint %}

#### :grapes:Gradient accumulation might inflate your losses

{% columns %}
{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FET1GgLeZanVHDpPXLkM9%2FTransformers%20%2B%20TRL%20%2B%20Gemma-4.png?alt=media&amp;token=0149149f-4d34-4bcb-a545-42e12d5127eb" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FZZola3h7ujfqz87VdnQm%2FUnsloth%20%2B%20Gemma-4.png?alt=media&amp;token=37fc2b61-ae5b-4203-b9a7-388439aefae5" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

If you see losses higher than 13-15 (like 100 or 300) most likely gradient accumulation is not being accounted properly - we have **fixed this as part of Unsloth and Unsloth Studio.**

To read more about gradient accumulation see our gradient accumulation bug fix blog: <https://unsloth.ai/blog/gradient>

#### :interrobang:IndexError on Gemma-4 31B and 26B-A4B inference

You might see this error when doing inference with 31B and 26B:

```python
File "/.../cache_utils.py", line 937, in update
    keys, values = self.layers[layer_idx].update(...)
IndexError: list index out of range
```

The culprit is below:

```python
if hasattr(decoder_config, "num_kv_shared_layers"):
    layer_types = layer_types[: -decoder_config.num_kv_shared_layers]
```

Where Gemma-4 31B and 26B-A4B ship with `num_kv_shared_layers = 0`. In Python, `-0 == 0`, so `layer_types[:-0]` collapses to `layer_types[:0] == []`. The cache is built with zero layer slots and the very first attention forward crashes inside `Cache.update`.

#### :no\_entry: `use_cache = True` generation was gibberish for E2B, E4B

[See issue](https://github.com/huggingface/transformers/issues/45242) "\[Gemma 4] `use_cache=False` corrupts attention computation, producing garbage logits #45242"

Gemma-4 E2B and E4B share KV state across layers (`num_kv_shared_layers = 20` and `18`). The cache is the only place where early layers stash KV for later layers to reuse. When `use_cache=False` (as every QLoRA tutorial sets, and as `gradient_checkpointing=True` forces), `Gemma4TextModel.forward` skips cache construction, so the KV-shared layers fall through to recomputing K and V locally from the current hidden states. The logits become garbage and training loss diverges.

**Before (`unsloth/gemma-4-E2B-it`, prompt "What is 1+1?"):**

```
use_cache=True  -> '1 + 1 = **2**'
use_cache=False -> 'BROAD\肯. Specificallyboard K supposed\_n통  \'
max_abs_logit_diff: 48.937500
```

**After our fix:**

```
use_cache=True  -> '1 + 1 = **2**'
use_cache=False -> '1 + 1 = **2**'
max_abs_logit_diff: 0.000000     (bit-exact parity, all 9 tokens identical)
```

#### :radio:Audio float16 overflow

`Gemma4AudioAttention` uses `config.attention_invalid_logits_value = -1e9` in a `masked_fill` call. On fp16 (Tesla T4), -1e9 overflows the fp16 max of 65504, causing:

```python
RuntimeError: value cannot be converted to type c10::Half without overflow
```

This was due to `self.config.attention_invalid_logits_value` :

```python
attn_weights = attn_weights.masked_fill(
    attention_mask.logical_not(), self.config.attention_invalid_logits_value
)
```

#### 💡Tips for Gemma-4

1. If you want to **preserve reasoning** ability, you can mix reasoning-style examples with direct answers (keep a minimum of 75% reasoning). Otherwise you can emit it fully.\
   \
   Use `gemma-4` for the non thinking chat-template and `gemma-4-thinking` for the thinking variant.\
   Use the thinking one for the larger 26B and 31B ones, and the non thinking one for the small ones.<br>

   ```python
   from unsloth.chat_templates import get_chat_template
   tokenizer = get_chat_template(
       tokenizer,
       chat_template = "gemma-4-thinking", # Or "gemma-4"
   )
   ```
2. To enable thinking mode, use `enable_thinking = True / False` in `tokenizer.apply_chat_template`<br>

   Thinking enabled:

   <pre class="language-python" data-overflow="wrap"><code class="lang-python">processor.tokenizer.apply_chat_template([
       {"role" : "user", "content" : "What is 2+2?"},
   ], tokenize = False, enable_thinking = True, add_generation_prompt = True)
   </code></pre>

   Will print `<bos><|turn>system\n<|think|><turn|>\n<|turn>user\nWhat is 2+2?<turn|>\n<|turn>model\n`<br>

   Thinking disabled:

   ```python
   processor.tokenizer.apply_chat_template([
       {"role" : "user", "content" : "What is 2+2?"},
   ], tokenize = False, enable_thinking = False, add_generation_prompt = True)
   ```

   Will print `<bos><|turn>user\nWhat is 2+2?<turn|>\n<|turn>model\n<|channel>thought\n<channel|>`
3. Gemma 4 is powerful for multilingual fine-tuning as it supports 140 languages.
4. It is recommended to train **E4B QLoRA** rather than **E2B LoRA** as the E4B is bigger and the quantization accuracy difference is miniscule. Gemma 4 E4B LoRA is even better.
5. After fine-tuning, you can export to [GGUF](#saving-export-your-fine-tuned-model) (for llama.cpp/Unsloth/Ollama/etc.)

### ⚡Quickstart

#### 🦥 Unsloth Studio Guide

{% columns %}
{% column %}
Gemma 4 can be run and fine-tuned in [Unsloth Studio](/docs/new/studio), our new open-source web UI for local AI.

With Unsloth Studio, you can run models locally on **MacOS, Windows**, Linux and train NVIDIA GPUs. Intel, MLX and AMD training support coming this month.
{% endcolumn %}

{% column %}

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpZlhqoILYOzznGpbudUk%2Funsloth%20studio%20gemma%20graphic.png?alt=media&amp;token=75e41585-e363-45cf-a87e-4d02960766ed" alt=""><figcaption></figcaption></figure></div>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

Run in your terminal:

**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows PowerShell:**

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% hint style="success" %}
**Installation will be quick and take approx 1-2 mins.**
{% endhint %}
{% endstep %}

{% step %}

#### Launch Unsloth

**MacOS, Linux, WSL and Windows:**

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

**Then open `http://localhost:8888` in your browser.**
{% endstep %}

{% step %}

#### Train Gemma 4

On first launch you will need to create a password to secure your account and sign in again later. You’ll then see a brief onboarding wizard to choose a model, dataset, and basic settings. You can skip it at any time.

Search for Gemma 4 in the search bar and select your desired model and dataset. Next, adjust your hyperparameters, context length as desired.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpZlhqoILYOzznGpbudUk%2Funsloth%20studio%20gemma%20graphic.png?alt=media&amp;token=75e41585-e363-45cf-a87e-4d02960766ed" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

#### Monitor training progress

After you click start training, you will be able to monitor and observe the training progress of the model. The training loss should be steadily decreasing.\
Once done, the model will be automatically saved.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FeBrnu9zxARIkhOHzd0pq%2FScreenshot%202026-04-07%20at%205.53.32%E2%80%AFAM.png?alt=media&amp;token=dae77231-5020-4e8c-b2b8-cc49a98a9edf" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

#### Export your fine-tuned model

Once done, Unsloth Studio allows you to export the model to GGUF, safetensor etc formats.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FBtpx58zCdrOD4zB4DPSC%2FScreenshot%202026-04-07%20at%206.12.41%E2%80%AFAM.png?alt=media&amp;token=05f05af2-5f7f-4b91-9c99-21d6a9b04935" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}

#### Compare fine-tuned model vs original model

Click on `Compare Mode` to compare the LoRA adapter and the original model.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fvm2CBSg7QBkKwTMKutyr%2FScreenshot%202026-04-07%20at%206.14.50%E2%80%AFAM.png?alt=media&amp;token=8c9c159f-9d5b-4468-8984-681d19ebc427" alt="" width="563"><figcaption></figcaption></figure></div>
{% endstep %}
{% endstepper %}

#### 🦥 Unsloth Core (code-based) Guide

We made free notebooks for Gemma 4:

| [E4B **(Inference + Text)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E4B\)-Text.ipynb) | [E4B **(Vision + Text)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E4B\)-Vision.ipynb) | [E4B **(Audio)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E4B\)-Audio.ipynb) |
| --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| [**31B** (Kaggle)](https://www.kaggle.com/code/danielhanchen/gemma4-31b-unsloth)                                                  | [E2B **(Vision + Text)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)-Vision.ipynb) | [E2B **(Audio)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)-Audio.ipynb) |

And for reinforcement learning (RL): [E2B **(RL GRPO)**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(E2B\)_Reinforcement_Learning_Sudoku_Game.ipynb)

We also made notebooks for the larger Gemma 4 models but they need A100:

| [Gemma-4-26B-A4B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(26B_A4B\)-Vision.ipynb) - A100 GPU | [Gemma-4-31B](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma4_\(31B\)-Vision.ipynb) - A100 GPU |
| --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |

{% hint style="info" %}
**If you'd like to do** [**GRPO**](/docs/get-started/reinforcement-learning-rl-guide)**, it works in Unsloth if you disable fast vLLM inference and use Unsloth inference instead. Follow our** [**Vision RL**](/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl) **notebook examples.**
{% endhint %}

Below is a standalone Gemma-4-26B-A4B-it text SFT recipe. This is text only - see also our [vision fine-tuning](/docs/basics/vision-fine-tuning) section for more details.

{% code expandable="true" %}

````python
from unsloth import FastModel
import torch

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/gemma-4-26B-A4B-it", # Change this to unsloth/gemma-4-E2B-it etc
    dtype = None, # None for auto detection
    max_seq_length = 8192, # Choose any for long context!
    load_in_4bit = True,  # 4 bit quantization to reduce memory
    full_finetuning = False, # [NEW!] We have full finetuning now!
    # token = "YOUR_HF_TOKEN", # HF Token for gated models
)

"""# Gemma 4 can process Text, Vision and Audio!

Let's first experience how Gemma 4 can handle multimodal inputs. We use Gemma 4's recommended settings of `temperature = 1.0, top_p = 0.95, top_k = 64`
"""

from transformers import TextStreamer
# Helper function for inference
def do_gemma_4_inference(messages, max_new_tokens = 128):
    _ = model.generate(
        **tokenizer.apply_chat_template(
            messages,
            add_generation_prompt = True, # Must add for generation
            tokenize = True,
            return_dict = True,
            return_tensors = "pt",
        ).to("cuda"),
        max_new_tokens = max_new_tokens,
        use_cache=True,
        temperature = 1.0, top_p = 0.95, top_k = 64,
        streamer = TextStreamer(tokenizer, skip_prompt = True),
    )

"""# Gemma 4 can see images!

<img src="https://files.worldwildlife.org/wwfcmsprod/images/Sloth_Sitting_iStock_3_12_2014/story_full_width/8l7pbjmj29_iStock_000011145477Large_mini__1_.jpg" alt="Alt text" height="256">
"""

sloth_link = "https://files.worldwildlife.org/wwfcmsprod/images/Sloth_Sitting_iStock_3_12_2014/story_full_width/8l7pbjmj29_iStock_000011145477Large_mini__1_.jpg"

messages = [{
    "role" : "user",
    "content": [
        { "type": "image", "image" : sloth_link },
        { "type": "text",  "text" : "Which films does this animal feature in?" }
    ]
}]
# You might have to wait 1 minute for Unsloth's auto compiler
do_gemma_4_inference(messages, max_new_tokens = 256)

"""Let's make a poem about sloths!"""

messages = [{
    "role": "user",
    "content": [{ "type" : "text",
                  "text" : "Write a poem about sloths." }]
}]
do_gemma_4_inference(messages)

"""# Let's finetune Gemma 4!

You can finetune the vision and text parts for now through selection - the audio part can also be finetuned - we're working to make it selectable as well!

We now add LoRA adapters so we only need to update a small amount of parameters!
"""

model = FastModel.get_peft_model(
    model,
    finetune_vision_layers     = False, # Turn off for just text!
    finetune_language_layers   = True,  # Should leave on!
    finetune_attention_modules = True,  # Attention good for GRPO
    finetune_mlp_modules       = True,  # Should leave on always!

    r = 8,           # Larger = higher accuracy, but might overfit
    lora_alpha = 8,  # Recommended alpha == r at least
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
)

"""<a name="Data"></a>
### Data Prep
We now use the `Gemma-4` format for conversation style finetunes. We use [Maxime Labonne's FineTome-100k](https://huggingface.co/datasets/mlabonne/FineTome-100k) dataset in ShareGPT style. Gemma-4 renders multi turn conversations like below:

```
<bos><|turn>user
Hello<turn|>
<|turn>model
Hey there!<turn|>
```
We use our `get_chat_template` function to get the correct chat template. We support `zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, phi3, llama3, phi4, qwen2.5, gemma3, gemma-4` and more.
"""

from unsloth.chat_templates import get_chat_template
tokenizer = get_chat_template(
    tokenizer,
    chat_template = "gemma-4-thinking",
)

"""We get the first 3000 rows of the dataset"""

from datasets import load_dataset
dataset = load_dataset("mlabonne/FineTome-100k", split = "train[:3000]")

"""We now use `standardize_data_formats` to try converting datasets to the correct format for finetuning purposes!"""

from unsloth.chat_templates import standardize_data_formats
dataset = standardize_data_formats(dataset)

"""Let's see how row 100 looks like!"""

dataset[100]

"""We now have to apply the chat template for `Gemma-3` onto the conversations, and save it to `text`. We remove the `<bos>` token using removeprefix(`'<bos>'`) since we're finetuning. The Processor will add this token before training and the model expects only one."""

def formatting_prompts_func(examples):
   convos = examples["conversations"]
   texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False).removeprefix('<bos>') for convo in convos]
   return { "text" : texts, }

dataset = dataset.map(formatting_prompts_func, batched = True)

"""Let's see how the chat template did! Notice there is no `<bos>` token as the processor tokenizer will be adding one."""

dataset[100]["text"]

"""<a name="Train"></a>
### Train the model
Now let's train our model. We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`.
"""

from trl import SFTTrainer, SFTConfig
trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    eval_dataset = None, # Can set up evaluation!
    args = SFTConfig(
        dataset_text_field = "text",
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4, # Use GA to mimic batch size!
        warmup_steps = 5,
        # num_train_epochs = 1, # Set this for 1 full training run.
        max_steps = 60,
        learning_rate = 2e-4, # Reduce to 2e-5 for long training runs
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.001,
        lr_scheduler_type = "linear",
        seed = 3407,
        report_to = "none", # Use TrackIO/WandB etc
    ),
)

"""We also use Unsloth's `train_on_completions` method to only train on the assistant outputs and ignore the loss on the user's inputs. This helps increase accuracy of finetunes!"""

from unsloth.chat_templates import train_on_responses_only
trainer = train_on_responses_only(
    trainer,
    instruction_part = "<|turn>user\n",
    response_part = "<|turn>model\n",
)

"""Let's verify masking the instruction part is done! Let's print the 100th row again.  Notice how the sample only has a single `<bos>` as expected!"""

tokenizer.decode(trainer.train_dataset[100]["input_ids"])

"""Now let's print the masked out example - you should see only the answer is present:"""

tokenizer.decode([tokenizer.pad_token_id if x == -100 else x for x in trainer.train_dataset[100]["labels"]]).replace(tokenizer.pad_token, " ")

"""# Let's train the model!

To resume a training run, set `trainer.train(resume_from_checkpoint = True)`
"""

trainer_stats = trainer.train()
````

{% endcode %}

{% hint style="info" %}
If you OOM:

* Drop `per_device_train_batch_size` to **1** and/or reduce `max_seq_length`.&#x20;
* Keep `use_`[`gradient_checkpointing`](/docs/blog/500k-context-length-fine-tuning#unsloth-gradient-checkpointing-enhancements)`="unsloth"` on (it’s designed to reduce VRAM use and extend context length).
  {% endhint %}

**Loader example for MoE (bf16 LoRA):**

```python
import os
import torch
from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Gemma-4-26B-A4B-it",
    max_seq_length = 2048,
    load_in_4bit = False,     # MoE QLoRA not recommended, dense 31B is fine
    load_in_16bit = True,     # bf16/16-bit LoRA
    full_finetuning = False,
)
```

Once loaded, you’ll attach LoRA adapters and train similarly to the SFT example above.

### Reinforcement Learning (RL)

You can now train Gemma 4 with RL, GSPO, GRPO etc with [our free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_\(4B\)_Vision_GRPO.ipynb).

{% columns %}
{% column %}
Gemma 4 E2B RL works on 9GB.

{% embed url="<https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision_GRPO.ipynb>" %}

The notebook's goal is to make Gemma 4 learn to solve Sudoku puzzles using [GRPO](/docs/get-started/reinforcement-learning-rl-guide#from-rlhf-ppo-to-grpo-and-rlvr).

The model will devise a strategy to fill in empty cells, and we'll reward it for correct placements and completing valid puzzles.

You can run Gemma 4 RL with Unsloth even though it is not supported by vLLM, by setting `fast_inference=False` when loading the model:
{% endcolumn %}

{% column %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fw08jKlXDLJji37JZPYNx%2Fgemma%204%20rl%20sodoku%20nb.png?alt=media&amp;token=379c607d-60b1-4b35-bcdd-bf6a86911f85" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-4-E2B-it",
    fast_inference=False,
)
```

### MoE fine-tuning (26B-A4B)

The **26B-A4B** model is the speed / quality middle ground in the Gemma 4 lineup. Since it is an **MoE** model with only a subset of parameters active per token, a conservative fine-tuning approach is:

* use **LoRA** rather than full fine-tuning
* prefer **16-bit / bf16 LoRA** if memory allows
* start with shorter contexts and smaller ranks first
* scale up only after the pipeline is stable

If your goal is the highest quality and you have more memory, use **31B** instead.

### Multimodal fine-tuning (E2B / E4B)

Because **E2B** and **E4B** support **image** and **audio**, they are the main Gemma 4 variants for multimodal fine-tuning.

* load the multimodal model with `FastVisionModel`
* keep `finetune_vision_layers = False` first
* fine-tune only the language, attention, and MLP layers
* enable vision or audio layers later if your task needs it

#### Gemma 4 Multimodal LoRA example:

{% code expandable="true" %}

````python
from unsloth import FastVisionModel # FastLanguageModel for LLMs
import torch

model, processor = FastVisionModel.from_pretrained(
    "unsloth/gemma-4-26B-A4B-it",
    load_in_4bit = True, # Use 4bit to reduce memory use. False for 16bit LoRA.
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
)

"""We now add LoRA adapters for parameter efficient fine-tuning, allowing us to train only 1% of all model parameters efficiently.

**[NEW]** We also support fine-tuning only the vision component, only the language component, or both. Additionally, you can choose to fine-tune the attention modules, the MLP layers, or both!
"""

model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers     = True, # False if not finetuning vision layers
    finetune_language_layers   = True, # False if not finetuning language layers
    finetune_attention_modules = True, # False if not finetuning attention layers
    finetune_mlp_modules       = True, # False if not finetuning MLP layers

    r = 32,                           # The larger, the higher the accuracy, but might overfit
    lora_alpha = 32,                  # Recommended alpha == r at least
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,               # We support rank stabilized LoRA
    loftq_config = None,               # And LoftQ
    target_modules = "all-linear",    # Optional now! Can specify a list if needed
)

"""<a name="Data"></a>
### Data Prep
We'll use a sampled dataset of handwritten math formulas. The objective is to convert these images into a computer-readable format—specifically LaTeX—so they can be rendered. This is particularly useful for complex expressions.

You can access the dataset [here](https://huggingface.co/datasets/unsloth/LaTeX_OCR). The full dataset is [here](https://huggingface.co/datasets/linxy/LaTeX_OCR).
"""

from datasets import load_dataset
dataset = load_dataset("unsloth/LaTeX_OCR", split = "train")

"""Let's take an overview of the dataset. We'll examine the second image and its corresponding caption."""

dataset

dataset[2]["image"]

dataset[2]["text"]

"""We can also render LaTeX directly in the browser!"""

from IPython.display import display, Math, Latex

latex = dataset[3]["text"]
display(Math(latex))

"""To format the dataset, all vision fine-tuning tasks should follow this format:

```python
[
    {
        "role": "user",
        "content": [
            {"type": "text", "text": instruction},
            {"type": "image", "image": sample["image"]},
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": instruction},
            {"type": "image", "image": sample["image"]},
        ],
    },
]
```
"""

instruction = "Write the LaTeX representation for this image."

def convert_to_conversation(sample):
    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": instruction},
                {"type": "image", "image": sample["image"]},
            ],
        },
        {"role": "assistant", "content": [{"type": "text", "text": sample["text"]}]},
    ]
    return {"messages": conversation}
pass

"""Let's convert the dataset into the "correct" format for finetuning:"""

converted_dataset = [convert_to_conversation(sample) for sample in dataset]

"""The first example is now structured like below:"""

converted_dataset[0]

"""Lets take the Gemma 4 instruction chat template and use it in our base model"""

from unsloth import get_chat_template

processor = get_chat_template(
    processor,
    "gemma-4-thinking"
)

"""Before fine-tuning, let us evaluate the base model's performance. We do not expect strong results, as it has not encountered this chat template before."""

image = dataset[2]["image"]
instruction = "Write the LaTeX representation for this image."

messages = [
    {
        "role": "user",
        "content": [{"type": "image"}, {"type": "text", "text": instruction}],
    }
]
input_text = processor.apply_chat_template(messages, add_generation_prompt = True)
inputs = processor(
    image,
    input_text,
    add_special_tokens = False,
    return_tensors = "pt",
).to("cuda")

from transformers import TextStreamer

text_streamer = TextStreamer(processor, skip_prompt = True)
result = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,
                        use_cache = True, temperature = 1.0, top_p = 0.95, top_k = 64)

"""You can see it's absolutely terrible! It doesn't follow instructions at all

<a name="Train"></a>
### Train the model
Now let's train our model. We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support `DPOTrainer` and `GRPOTrainer` for reinforcement learning!!

We use our new `UnslothVisionDataCollator` which will help in our vision finetuning setup.
"""

from unsloth.trainer import UnslothVisionDataCollator
from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model = model,
    train_dataset = converted_dataset,
    processing_class = processor.tokenizer,
    data_collator = UnslothVisionDataCollator(model, processor),
    args = SFTConfig(
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        max_grad_norm = 0.3,
        warmup_ratio = 0.03,
        max_steps = 60,
        # num_train_epochs = 2, # Set this instead of max_steps for full training runs
        learning_rate = 2e-4,
        logging_steps = 1,
        save_strategy = "steps",
        optim = "adamw_8bit",
        weight_decay = 0.001,
        lr_scheduler_type = "cosine",
        seed = 3407,
        output_dir = "outputs",
        report_to = "none", # For Weights and Biases or others

        # You MUST put the below items for vision finetuning:
        remove_unused_columns = False,
        dataset_text_field = "",
        dataset_kwargs = {"skip_prepare_dataset": True},
        max_length = 2048,
    )
)

trainer_stats = trainer.train()
````

{% endcode %}

#### Image example format

Remember: for Gemma 4 multimodal prompts, put the image **before** the text instruction.

{% code expandable="true" %}

```json
{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image", "image": "/path/to/image OR object"},
        {"type": "text", "text": "Extract all text from this receipt. Return line items, total, merchant, and date as JSON."}
      ]
    },
    {
      "role": "assistant",
      "content": [
        {"type": "text", "text": "{\"merchant\": \"Example Store\", \"total\": \"19.99\"}"}
      ]
    }
  ]
}
```

{% endcode %}

#### Audio example format

Audio is for **E2B / E4B** only. Keep clips short and task-specific.

{% code expandable="true" %}

```json
{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "audio", "audio": "/path/to/audio OR object"},
        {"type": "text", "text": "Transcribe the following speech segment in English into English text. Only output the transcription."}
      ]
    },
    {
      "role": "assistant",
      "content": [
        {"type": "text", "text": "Hello everyone and welcome back."}
      ]
    }
  ]
}
```

{% endcode %}

### Saving / export fine-tuned model

You can view our specific inference / deployment guides for [Unsloth Studio](/docs/new/studio/export), [llama.cpp](/docs/basics/inference-and-deployment/saving-to-gguf), [vLLM](/docs/basics/inference-and-deployment/vllm-guide), [llama-server](/docs/basics/inference-and-deployment/llama-server-and-openai-endpoint), [Ollama](/docs/basics/inference-and-deployment/saving-to-ollama) or [SGLang](/docs/basics/inference-and-deployment/sglang-guide).

#### Save to GGUF

Unsloth supports saving directly to GGUF:

```python
model.save_pretrained_gguf("directory", tokenizer, quantization_method = "q4_k_m")
model.save_pretrained_gguf("directory", tokenizer, quantization_method = "q8_0")
model.save_pretrained_gguf("directory", tokenizer, quantization_method = "f16")
```

Or push GGUFs to Hugging Face:

```python
model.push_to_hub_gguf("hf_username/directory", tokenizer, quantization_method = "q4_k_m")
model.push_to_hub_gguf("hf_username/directory", tokenizer, quantization_method = "q8_0")
```

If the exported model behaves worse in another runtime, Unsloth flags the most common cause: **wrong chat template / EOS token at inference time** (you must use the same chat template you trained with).

For more details read our inference guides:

{% columns %}
{% column width="50%" %}
{% content-ref url="/pages/gEugERiAw2ztDNt98JVR" %}
[Inference & Deployment](/docs/basics/inference-and-deployment)
{% endcontent-ref %}

{% content-ref url="/pages/T7ZPf3SNAwDykZNgXptE" %}
[GGUF & llama.cpp](/docs/basics/inference-and-deployment/saving-to-gguf)
{% endcontent-ref %}
{% endcolumn %}

{% column width="50%" %}
{% content-ref url="/pages/5ZU2kPF2eJ7VK0GeEUhu" %}
[Model Export](/docs/new/studio/export)
{% endcontent-ref %}

{% content-ref url="/pages/fhJtaLFFXVsGnbMUiACo" %}
[vLLM](/docs/basics/inference-and-deployment/vllm-guide)
{% endcontent-ref %}
{% endcolumn %}
{% endcolumns %}

### Gemma 4 data best practices

Gemma 4 has a few formatting details you need to keep in mind.

#### 1. Use standard chat roles

Gemma 4 uses the standard:

* `system`
* `user`
* `assistant`

This means your SFT dataset should be written in regular chat format rather than older Gemma-specific role formats.

#### 2. Thinking mode is explicit

If you want to preserve thinking-style behavior during SFT:

* keep the format consistent
* decide whether you want to train on **visible thought blocks** or on **final answers only**
* do **not** mix multiple incompatible thought formats in the same dataset

For most production assistants, the simplest setup is to fine-tune on the **final visible answer only**.

#### 3. Multi-turn rule

For multi-turn conversations, only keep the **final visible answer** in the conversation history. Do **not** feed earlier thought blocks back into later turns.




---

[Next Page](https://unsloth.ai/docs/llms-full.txt/1)