> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/models/inkling-2-mini.md).

# Inkling 2 Mini - Run Locally Guide

Learn how to run Thinking Machines Lab’s Inkling 2 Mini locally with Unsloth and llama.cpp.

**Inkling 2 Mini** is Thinking Machines Lab’s **26.4B MoE (5.4B active)** model for coding, agentic, and **vision** workloads. Released under Apache 2.0 and built on a Looped Transformers variant, it has **1M context** and scores **48.2% on DeepSWE v1.1** and **77.9% on SWEBench Pro v2 (Hard)**, making it the strongest for its size, ahead of Qwen3.8-27B. It ties Qwen3.8-27B on Terminal Bench 4.0 at **5.6%**.

This guide covers running Inkling Mini on a Mac, NVIDIA or AMD device with Unsloth Desktop or llama.cpp. Run on 16GB RAM or **24GB RAM** for **full precision**.  GGUF: [**Inkling 2 Mini GGUF**](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF)

<a href="https://unsloth.ai/docs/models/inkling-2-mini.md#run-inkling-2-mini-guide" class="button primary">Run Inkling 2 Mini Guide</a><a href="https://unsloth.ai/download" class="button secondary">Download Unsloth</a>

### ⚙️ Usage Guide

#### Inkling 2 Mini Requirements

**`UD-Q8_K_XL`** is **full precision**. The recommended quant is **`UD-Q4_K_XL` (17.6 GB).**

**Table: Hardware Requirements** (RAM + VRAM, or unified memory)

| 1-bit | 2-bit    | 3-bit    | 4-bit    | 8-bit (Lossless) |
| ----- | -------- | -------- | -------- | ---------------- |
| 9 GB  | 10-11 GB | 13-15 GB | 17-18 GB | 24 GB            |

See [Quantization Analysis](#quantization-analysis) for all GGUF sizes and results.

{% hint style="success" %}
Use these ranges as a starting point. Longer contexts, image inputs and other apps can use more memory. The `mmproj-F16.gguf` vision projector adds 639 MB to the download.
{% endhint %}

#### Recommended Settings

| Temperature | Top-p | Top-k | Min-p |
| ----------- | ----- | ----- | ----- |
| `1.0`       | `1.0` | `0`   | `0.0` |

The model card uses **32K context** and the embedded chat template in its llama.cpp example. Reduce context length if memory is limited. [Model card settings](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF#running)

## Run Inkling 2 Mini Guide

Use an Unsloth or llama.cpp build with **looped-layer support** and **per-expert NVFP4 scales**. [Inkling 2 Mini model card](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF)

<a href="https://unsloth.ai/docs/models/inkling-2-mini.md#run-inkling-2-mini-in-unsloth-desktop" class="button primary">Run in Unsloth Desktop</a><a href="https://unsloth.ai/docs/models/inkling-2-mini.md#run-inkling-2-mini-in-llama-cpp" class="button secondary">Run in llama.cpp</a>

### 🦥 Run Inkling 2 Mini in Unsloth Desktop <a href="#run-inkling-2-mini-in-unsloth-desktop" id="run-inkling-2-mini-in-unsloth-desktop"></a>

{% stepper %}
{% step %}

#### Install Unsloth

Download and launch the [Unsloth Desktop app](https://unsloth.ai/docs/desktop.md) for macOS, Windows or Linux.

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

* <i class="fa-apple">:apple:</i> [Download for macOS](https://unsloth.ai/download/mac)
* <i class="fa-windows">:windows:</i> [Download for Windows](https://unsloth.ai/download/windows)
* <i class="fa-linux">:linux:</i> [Download for Linux](https://unsloth.ai/download/linux)

Or, if you prefer to install manually:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Search and download Inkling 2 Mini

In **Chat**, open **Select model**, or go to **Model hub**. Search for **Inkling 2 Mini**, select [**unsloth/Inkling-2-Mini-GGUF**](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF) and download the quant that fits your device. Once it downloads, select the model in Chat and start a conversation. You can also load a compatible Mini GGUF from your local files.
{% endstep %}

{% step %}

#### Run and Serve Inkling 2 Mini with the Unsloth API

This example uses the 4-bit `UD-Q4_K_XL` quant (16.61 GB). Change the suffix after the colon to select another quant from the repository.

{% code overflow="wrap" %}

```bash
unsloth run --model unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0
```

{% endcode %}

Use the endpoint URL and API key printed in the terminal to connect your client. [Unsloth API guide](https://unsloth.ai/docs/basics/api.md)
{% endstep %}
{% endstepper %}

### 🦙 Run Inkling 2 Mini in llama.cpp <a href="#run-inkling-2-mini-in-llama-cpp" id="run-inkling-2-mini-in-llama-cpp"></a>

{% stepper %}
{% step %}

#### Install llama.cpp

Use a prebuilt `llama-server` with both features above (use unslothai/llama.cpp prebuilts), or check out a source revision with PR XXXXX into `llama.cpp` and build it. Install Git, CMake and a C++ compiler first.

On Ubuntu or WSL, install the build dependencies.

```bash
sudo apt-get update
sudo apt-get install -y git cmake build-essential libssl-dev libcurl4-openssl-dev
```

For NVIDIA, install the [CUDA toolkit](https://developer.nvidia.com/cuda-downloads). On Apple Silicon, set `-DGGML_CUDA=OFF` to use Metal. [llama.cpp build instructions](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev git -y
git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp fetch origin pull/32187/head:inkling2
git -C llama.cpp checkout inkling2
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

{% endstep %}

{% step %}

#### Download the GGUF

Download the 4-bit `UD-Q4_K_XL` GGUF and the F16 vision projector. For another quant, use its filename from the repository. [GGUF files](https://huggingface.co/unsloth/Inkling-2-Mini-GGUF/tree/main)

```bash
pip install -U huggingface_hub
hf download unsloth/Inkling-2-Mini-GGUF \
    Inkling-2-Mini-UD-Q4_K_XL.gguf mmproj-F16.gguf \
    --local-dir Inkling-2-Mini-GGUF
```

If the repository is private, run `hf auth login` with an account that has access before downloading.
{% endstep %}

{% step %}

#### Run Inkling 2 Mini

Start the local chat server. If you installed a prebuilt binary, use its path for `llama-server`.

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    --model Inkling-2-Mini-GGUF/Inkling-2-Mini-UD-Q4_K_XL.gguf \
    --mmproj Inkling-2-Mini-GGUF/mmproj-F16.gguf \
    --jinja --ctx-size 32768 --n-gpu-layers 99 --parallel 1 \
    --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
    --host 127.0.0.1 --port 8080
```

{% endcode %}

Open <http://127.0.0.1:8080> to chat or upload an image. For text-only chat, omit the projector from the download and drop `--mmproj` from the server command. If memory is limited, reduce `--ctx-size` or `--n-gpu-layers`.
{% endstep %}
{% endstepper %}

### :play\_pause:Looped transformer architecture

Inkling-2-Mini uses a modified looped transformer architecture where layers 6-17 are looped 3 times and if we combine this with embeddings, only 5.4B gets activated. This is different from other LLM architectures where each decoder layer is only run once - Inkling-2-Mini essentially runs layers 6 to 17 three times, effectively compressing its disk space and footprint during the training process.

It is rumored that the GPT-6 Astra class of models might also employ the looped transformer architecture as well, and Inkling-2-Mini is the first powerful open model to employ a variant of this mechanism.

### 📊 Quantization Analysis <a href="#quantization-analysis" id="quantization-analysis"></a>

We compare each GGUF with `UD-Q8_K_XL` across seven domains using identical llama.cpp settings. The reference keeps the original NVFP4 expert weights and stores the remaining weights in BF16. UD-Q4\_K\_XL is also nearly lossless with MXFP8 weights dequantized to Q8\_0. UD-Q8\_K\_XL upcasts MXFP8 to BF16 so KLD = 0.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FqWPOI9paCJed36b9qV1x%2Fsize_vs_top1(7).png?alt=media&amp;token=186fd174-4199-4d66-949e-fc851aa614ef" alt="" width="563"><figcaption></figcaption></figure>

The 4-bit `UD-Q4_K_XL` gets **0.0107 mean KLD** and **98.0% top-token agreement** at **17.63 GB**. The 2-bit `UD-IQ2_XXS` brings the download down to **8.49 GB** with **90.6% agreement**.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FObpB27Pzg7eddzGOefoK%2Fsize_vs_kld(4).png?alt=media&amp;token=b388d982-caaa-402f-bc8f-f49938036713" alt="" width="563"><figcaption></figcaption></figure>

| Quant        | Size (GB) | Mean KLD vs UD-Q8\_K\_XL | Same top token |
| ------------ | --------- | ------------------------ | -------------- |
| UD-IQ1\_S    | 7.76      | 0.0911                   | 89.4%          |
| UD-IQ1\_M    | 8.02      | 0.0867                   | 90.2%          |
| UD-IQ2\_XXS  | 8.49      | 0.0726                   | 90.6%          |
| UD-Q2\_K\_XL | 9.14      | 0.0595                   | 91.6%          |
| UD-IQ3\_XXS  | 11.66     | 0.0269                   | 94.7%          |
| UD-Q3\_K\_XL | 13.63     | 0.0174                   | 96.3%          |
| UD-IQ4\_XS   | 15.73     | 0.0109                   | 97.7%          |
| UD-Q4\_K\_M  | 16.61     | 0.0108                   | 97.9%          |
| UD-Q4\_K\_XL | 17.63     | 0.0107                   | 98.0%          |
| UD-Q8\_K\_XL | 22.87     | reference                |                |

### 📊 Benchmarks

In Thinking Machines Lab’s evaluations, Mini leads the three similarly sized models below on the listed software engineering and search benchmarks. Mini scores **6.0 percentage points higher than Qwen3.8-27B on DeepSWE v1.1** and **4.1 points higher on SWEBench Pro v2 (Hard)**.

| Benchmark                          | Inkling 2 Mini   | Gemma 4 31B  | Muse Glimmer | Qwen3.8-27B  | Nemotron 3.5 Super VL |
| ---------------------------------- | ---------------- | ------------ | ------------ | ------------ | --------------------- |
| Parameters (active / total B)      | 5 / 26           | 31 / 31      | 30 / 30      | 27 / 27      | 12 / 125              |
| SWE-bench Verified                 | **79.2%**        | 46.1%        | 76.0%        | 77.8%        | 71.1%                 |
| DeepSWE v1.1                       | **48.2%**        | 0.3%         | 4.4%         | 42.2%        | 24.7%                 |
| Terminal Bench v4.0                | **5.6%**         | 0.0%         | 0.5%         | **5.6%**     | 0.0%                  |
| AA Index v5.0                      | **20.1**         | 4.4          | 6.3          | **20.1**     | 13.6                  |
| Vals Index v2.2                    | **46.5%**        | –            | –            | –            | 29.8%                 |
| SWE-bench Pro (Hard)               | **77.9%**        | 14.2%        | 44.3%        | 73.8%        | 44.1%                 |
| Terminal Bench v2.1                | 71.6%            | 43.4%        | 51.7%        | **73.0%**    | 60.7%                 |
| BrowseComp                         | **79.0%**        | 37.0%        | 63.9%        | 61.6%        | 78.9%                 |
| DeepSearchQA                       | **85.3%**        | 61.7%        | 74.6%        | 79.7%        | 77.7%                 |
| GDPVal-AA v2.1                     | 1319             | 627          | 792          | **1427**     | 1162                  |
| AutomationBench-AA                 | 41.8%            | 4.9%         | 6.8%         | **48.2%**    | 14.8%                 |
| JobBench                           | **49.6%**        | 8.2%         | 16.1%        | 33.4%        | 36.5%                 |
| MMMU Pro                           | 74.9%            | 73.4%        | 74.3%        | **76.3%**    | 72.3%                 |
| OmniDocBench v1.6                  | **92.8%**        | 82.5%        | 90.4%        | 91.5%        | 77.9%                 |
| ScreenSpot Pro (original / Python) | **76.5 / 87.7%** | 5.9 / –      | 75.4 / –     | 57.6 / 78.9% | 43.5 / –              |
| Chartography (original / Python)   | **22.1 / 72.5%** | 12.0 / 11.8% | 17.4 / 25.1% | 12.2 / 52.1% | 9.7 / 29.3%           |
| HLE-Diamond                        | 15.2%            | 7.2%         | 9.1%         | **15.3%**    | 8.8%                  |
| ARC-AGI (v1 / v2)                  | **84.5 / 49.3%** | –            | –            | 87.5 / 42.4% | 22.7 / 1.5%           |
| LongBench v2                       | **66.9%**        | 52.9%        | 33.1%        | 52.3%        | 62.1%                 |
| AA-Omniscience (score)             | -8.6             | -48.5        | -32.2        | -6.4         | **-2.9**              |
| AA-Omniscience (non-hallucination) | 63.4%            | 14.5%        | 16.4%        | **71.4%**    | 69.5%                 |
| FORTRESS (adversarial)             | 89.5%            | 67.8%        | **90.3%**    | 78.8%        | 79.6%                 |
| FORTRESS (benign)                  | 96.4%            | 97.6%        | 89.1%        | **97.8%**    | 92.7%                 |
| StrongREJECT                       | 97.8%            | 98.1%        | 99.0%        | 99.0%        | **99.4%**             |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/models/inkling-2-mini.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
