> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/models/qwen3.8.md).

# Qwen3.8 - How to Run Locally

Qwen3.8 is Qwen's new family of models, including **Qwen3.8-27B**, Qwen3.8-**2.4T-A95B** and Qwen3.8**-Max**. Qwen3.8 has vision and thinking capabilities, a 256K context window (up to 1M tokens). Qwen3.8-27B the upcoming 27B parameter model will be released this Friday. Qwen3.8-2.4T-A95B is a 2.4T parameter open-weight model (95B active) that rivals GPT-5.6 Sol.

Full-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit [Unsloth](https://github.com/unslothai/unsloth) Dynamic UD-IQ1\_XXXS GGUFs takes **397GB (91% smaller)**, and dynamic IQ1\_S takes 508GB (89% smaller). Qwen3.8-27B will run locally on 16GB+ VRAM/RAM setups. You can run Qwen3.8 via [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop) or llama.cpp. [Qwen3.8-2.4T-A95B-GGUF](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF)

<a href="/pages/CLyZKmpoJZpdaJXhLW4v#run-qwen3.8-guide" class="button primary">Run Qwen3.8 Tutorials</a>

For **lossless** Qwen3.8, use BF16 (`BF16`), which is 4.9TB in disk space. Q8\_0 is 8-bit and needs 2.6TB.\
**Table: Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

| Dynamic 1-bit XXXS | Dynamic 1-bit Standard | Dynamic 2-bit | Q8\_0  | BF16 (Lossless) |
| ------------------ | ---------------------- | ------------- | ------ | --------------- |
| 397GB              | 508GB                  | 657 GB        | 2.6 TB | 4.9 TB          |

### New 1-bit data-types

We extended IQ1\_S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization (PTQ) without the need for QAT or QAD (quantization aware training / distillation)

Due to naming issues, we used TQ2\_0, TQ1\_0 and Q1\_0 otherwise it won't pop up in the HF repo.

<table><thead><tr><th>Dtype</th><th width="147.60000610351562">Naming</th><th width="114.39996337890625" align="right">BPW</th><th width="106.20001220703125" align="right"># entries</th><th width="112.79998779296875" align="right">Index bits</th><th width="106.4000244140625" align="right">Block</th></tr></thead><tbody><tr><td>IQ1_S</td><td>IQ1_S</td><td align="right"><strong>1.5625</strong></td><td align="right">2048</td><td align="right">11</td><td align="right">50 B</td></tr><tr><td>UD-IQ1_XS</td><td>TQ2_0</td><td align="right">1.4375</td><td align="right">1024</td><td align="right">10</td><td align="right">46 B</td></tr><tr><td>UD-IQ1_XXS</td><td>TQ1_0</td><td align="right">1.3125</td><td align="right">512</td><td align="right">9</td><td align="right">42 B</td></tr><tr><td>UD-IQ1_XXXS</td><td>Q1_0</td><td align="right"><strong>1.1875</strong></td><td align="right">256</td><td align="right">8</td><td align="right">38 B</td></tr></tbody></table>

We are still running benchmarks for the new data-types, but for other large models, we get **good results without any QAT / QAD**:

| Dtype        |     GiB |      PPL |      KLD |  top-p |
| ------------ | ------: | -------: | -------: | -----: |
| IQ1\_S       | 553.204 | 2.578876 | 0.564553 | 78.882 |
| UD-IQ1\_XS   | 513.583 | 2.931261 | 0.690161 | 75.726 |
| UD-IQ1\_XXS  | 473.961 | 3.540383 | 0.876007 | 71.284 |
| UD-IQ1\_XXXS | 434.340 | 4.488796 | 1.109944 | 66.257 |

### :gear: Usage Guide

Open Qwen3.8 models are **thinking-only**, while Qwen3.8-max is hybrid.

| Default                 |
| ----------------------- |
| temperature = 1.0       |
| top\_p = 0.95           |
| top\_k = 20             |
| min\_p = 0.0            |
| presence\_penalty = 0.0 |

* Context length = up to `1,010,000`
* `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`
* Low, High, Max Thinking can be toggled in Unsloth

If the model fits, you will get \~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading.

## Run Qwen3.8 Guide

You can now run Qwen3.8 in [llama.cpp](#run-in-llama.cpp) and Unsloth Studio. We will be utilizing the 397GB IQ1\_XXXS quant (named Q1\_0) for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type. GGUF: [**Qwen3.8-GGUF**](https://huggingface.co/unsloth/Kimi-K3-GGUF)

### 🦥 Run Qwen3.8 in Unsloth Desktop

Qwen3.8 can run in [Unsloth Desktop](#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:

{% columns %}
{% column %}

* Search, download, [run GGUFs](/docs/new/studio.md#run-models-locally) and safetensor models
* [**Self-healing** tool calling](/docs/new/studio.md#execute-code--heal-tool-calling) + **web search**
* [**Code execution**](/docs/new/studio.md#run-models-locally) (Python, Bash)
* [Automatic inference](/docs/new/studio.md#model-arena) parameter tuning (temp, top-p, etc.)
* Fast CPU + GPU inference via llama.cpp
* [Train LLMs](/docs/new/studio.md#no-code-training) 2x faster with 70% less VRAM
  {% endcolumn %}

{% column %}

<figure><img src="/files/0xoUQOYpTX661nFuPLSL" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% stepper %}
{% step %}

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop.md). Works on [macOS](/docs/get-started/install/mac.md), [Windows](/docs/get-started/install/windows-installation.md), and [Linux](/docs/get-started/install/linux.md).

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

MacOS, Linux, WSL:

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

Windows PowerShell:

```bash
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}

#### Search and download Qwen3.8

Go to [Unsloth Chat](/docs/new/studio/chat.md) or Model hub and search for Qwen 3.8 in the search bar and download your desired model and quant.

<figure><img src="/files/v00rFyy1hH2c9YenLaum" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run Qwen3.8

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat.md).

{% stepper %}
{% step %}
We need to use the specific IQ1\_XXXS branch [here](https://github.com/unslothai/llama.cpp/pull/61). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch iq1-narrow https://github.com/unslothai/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}
If you just want to run the standard IQ1\_S and other quants, then compile llama.cpp normally:

{% code overflow="wrap" %}

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endcode %}
{% endstep %}

{% step %}
Download the model via (after installing `pip install huggingface_hub`). You can choose `Q1_0` for `IQ1_XXXS` or other quantized versions like `Q8_0` . If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging.md)

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --local-dir unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --include "*Q1_0*" # Use "*IQ2_XXS*" for 2-bit
```

{% endstep %}

{% step %}
To run the model in llama-cli, do:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-2.4T-A95B-GGUF/UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
```

{% endcode %}
{% endstep %}

{% step %}
To run the general UD-IQ1\_S you can do:

{% code overflow="wrap" %}

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --local-dir unsloth/Qwen3.8-2.4T-A95B-GGFF \
    --include "*IQ1_S*" # Use "*IQ2_XXS*" for 2-bit
```

{% endcode %}
{% endstep %}

{% step %}
Then to run it:

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-2.4T-A95B-GGUF/UD-IQ1_S/Qwen3.8-2.4T-A95B-UD-IQ1_S-00001-of-00012.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
```

{% endcode %}
{% endstep %}
{% endstepper %}
{% endstep %}
{% endstepper %}

### 📊 Benchmarks

You can view further below for benchmarks in table format:

<figure><img src="/files/rUwPDAu4KgLzJCPv1VSJ" alt=""><figcaption></figcaption></figure>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://unsloth.ai/docs/models/qwen3.8.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.