> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/models/inkling.md). # Inkling - How to Run Locally Inkling by Thinking Machines Labs are open-weight multimodal models consisting of: **Inkling-Small** the new **276B** (12B active) model and the earlier **975B** (41B) parameter model. Licensed under Apache 2.0, the models support a 1M token context window, native multimodal input for **text**, **image**, and **audio**, and generates **text** as the output. Inkling excel at coding, agentic and tool-calling, RAG, chat, multilingual, and multimodal workloads. Inkling-Small dynamic 3-bit runs on a **128GB RAM device** (NVIDIA DGX Spark or Mac) and the smaller 4-bit runs on 132GB RAM. The BF16 weights for Inkling 975B requires **1.9 TB of disk space** - our dynamic GGUFs lowers this to **270–285 GB at dynamic 1-bit** and **317 GB at 2-bit**. [**Dynamic 1-bit retains \~74.2%**](#quantization-analysis) of top-1% accuracy whilst being **86% smaller!** You can run both models in [Unsloth Studio](#run-inkling-in-unsloth-studio) or llama.cpp. For Inkling-Small, dynamic 3-bit fits in 98GB from 544GB BF16 (82% smaller), and retains 73% of all accuracy. Dynamic 2-bit reaches 81% top-1% accuracy whilst being 82% smaller. This shows if we shrink the model down by 82% with our [Unsloth Dyanmic ](/docs/basics/unsloth-dynamic-2.0-ggufs.md)GGUF method - it does NOT mean the model gets 82% "dumber" - only \~18% degradation is seen. Thanks Thinking Machines Lab (TML) for giving Unsloth day-zero access. [**Inkling-Small-GGUF**](https://huggingface.co/unsloth/Inkling-Small-GGUF) and [**Inkling-GGUF**](https://huggingface.co/unsloth/inkling-GGUF) Run Inkling TutorialsQuantization Results #### :gear: Usage Guide {% columns %} {% column width="50%" %} The 1-bit dynamic quant `UD-IQ1_S` uses **270GB** of disk space - this will need a Mac Studio Ultra or machines with at least RAM+VRAM to be 290GB or so. The **1-bit** quant will fit on a 290GB RAM and 6/8-bit requires 900GB RAM. See the 1-bit GGUF in action to the right where we asked Inkling to make a Sudoku HTML game. See right for 1-bit Inkling GGUF in Unsloth: {% endcolumn %} {% column width="50%" %}
{% endcolumn %} {% endcolumns %} **Table: Inference hardware requirements** (units = total memory: RAM + VRAM, or unified memory): | Model | 2-bit | 3-bit | 4-bit | 6/8bit | BF16 | | ------------- | ------ | ------ | ------------ | ------ | ------- | | Inkling-Small | 89 GB | 128 GB | 132 - 170 GB | 256 GB | 543 GB | | Inkling | 325 GB | 450 GB | 600 GB | 870 GB | 1900 GB | **Table: Top-1% Retained (Accuracy Recovery) for 975B Inkling:** | 1-bit | 2-bit | 3-bit | 4-bit | 6/8bit | BF16 | | ----------- | ----- | ----- | ----- | ------ | ------ | | 74.2%-77.4% | 81.0% | 88.7% | 94.4% | 99.8% | 100.0% | {% hint style="success" %} For best performance, make sure your total available memory, including VRAM and system RAM, exceeds the quantized model file size by a comfortable margin. {% endhint %}

Sodoku with Inkling 1-bit

Audio parsing with 1-bit Inkling

### Recommended Settings Inkling has Non-thinking and Thinking modes. In [Unsloth Studio](#run-glm-5.2-in-unsloth-studio) you can easily toggle Thinking and non-Thinking with a UI. Use these settings for most use cases: | Default Settings (Most Tasks) | | ----------------------------- | | `temperature` = 1.0 | | `top_p` = 1.0 (Off) | * **Maximum context window:** `1,048,576`. #### Chat Template & thinking efforts Inkling uses an interesting "Thinking effort level:" system prompt component where it's a number from 0.00 to 0.99! Tokenizing the below: {% code overflow="wrap" expandable="true" %} ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("unsloth/inkling") tokenizer.apply_chat_template([ {"role" : "user", "content" : "What is 1+1?"}, {"role" : "assistant", "content" : "2!"}, {"role" : "user", "content" : "What is 2+2?"}, ], tokenize = False, reasoning_effort = "xhigh") ``` {% endcode %} will get the below: {% code overflow="wrap" expandable="true" %} ``` <|message_system|><|content_text|>Thinking effort level: 0.99<|end_message|><|message_user|><|content_text|>What is 1+1?<|end_message|><|message_model|><|content_text|>2!<|end_message|><|content_model_end_sampling|><|message_user|><|content_text|>What is 2+2?<|end_message|> ``` {% endcode %} #### Disabling Thinking, changing reasoning effort Inkling uses reasoning by default. It also supports reasoning efforts where `reasoning_effort` can be none = 0, low = 0.2, medium = 0.7, high = 0.9, xhigh = 0.99 and max = 0.99. To disable thinking, use `--chat-template-kwargs '{"reasoning_effort":'none'}'`. If you're on **Windows** Powershell, use: `--chat-template-kwargs "{\"reasoning_effort\":'none'}"` For reasoning effort customization and or to disable reasoning, use the below examples: {% code overflow="wrap" expandable="true" %} ```bash --chat-template-kwargs '{"reasoning_effort":"none"}' --chat-template-kwargs '{"reasoning_effort":"low"}' --chat-template-kwargs '{"reasoning_effort":"medium"}' --chat-template-kwargs '{"reasoning_effort":"high"}' --chat-template-kwargs '{"reasoning_effort":"xhigh"}' --chat-template-kwargs '{"reasoning_effort":"max"}' ``` {% endcode %} {% columns %} {% column %} Inkling also does interleaved thinking and tool calling - even with the dynamic 1-bit quant! {% endcolumn %} {% column %}
{% endcolumn %} {% endcolumns %} ## Run Inkling Tutorials: You can now run Inkling-Small and Inkling 975B in [llama.cpp](#run-in-llama.cpp) and [Unsloth Studio](https://unsloth.ai/docs/models/pages/cZiZ00cwk1GGz1O1TOHo#run-glm-5.2-in-unsloth-studio). For Inkling-Small we will be utilizing the 4-bit quant which fits on a 128GB Mac device. For Inkling 975B we will be utilizing the 270GB [`UD-IQ1_M`](https://huggingface.co/unsloth/inkling-GGUF) quant for best results in terms of accessibility and accuracy. ### 🦥 Run Inkling in Unsloth Studio Inkling-Small and Inkling can run in [Unsloth Studio](/docs/new/studio.md), an open-source web UI for local AI. **Unsloth Studio automatically offloads to RAM and detects multiGPU setups**. With Unsloth Studio, you can run models locally on **MacOS, Windows**, Linux and: {% columns %} {% column %} * Search, download, [run GGUFs](/docs/new/studio.md#run-models-locally) and safetensor models * [**Self-healing** tool calling](/docs/new/studio.md#execute-code--heal-tool-calling) + **web search** * [**Code execution**](/docs/new/studio.md#run-models-locally) (Python, Bash) * [Automatic inference](/docs/new/studio.md#model-arena) parameter tuning (temp, top-p, etc.) * Fast CPU + GPU inference via llama.cpp * [Train LLMs](/docs/new/studio.md#no-code-training) 2x faster with 70% less VRAM {% endcolumn %} {% column %}

1-bit Inkling GGUF multimodal example

{% endcolumn %} {% endcolumns %} {% stepper %} {% step %} **Install and Launch Unsloth** To install, run in your terminal: MacOS, Linux, WSL: ```bash curl -fsSL https://unsloth.ai/install.sh | sh ``` Windows PowerShell: ```bash irm https://unsloth.ai/install.ps1 | iex ``` **Launch Unsloth** MacOS, Linux, WSL and Windows: ```bash unsloth studio -H 0.0.0.0 -p 8888 ``` Then open `http://127.0.0.1:8888` (or your specific URL) in your browser. **Launch Unsloth securely with HTTPS and Cloudflare** **NEW!** Unsloth now provides a secure way to launch Studio over HTTPS through a free Cloudflare tunnel. Use the below (works in Windows, Mac & Linux): ```bash unsloth studio --secure ``` {% endstep %} {% step %} **Search and download** Inkling Unsloth Studio automatically offloads to RAM and detects multiGPU setups. On first launch you will need to create a password to secure your account and sign in again later. Then go to the [Unsloth Chat](/docs/new/studio/chat.md) tab and search for **Inkling** in the search bar and download your desired model and quant. Ensure you have enough compute the run the model.
{% endstep %} {% step %} **Run Inkling** Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings. For more information, you can view our [Unsloth Studio inference guide](/docs/new/studio/chat.md).

1-bit Inkling running in Unsloth Studio

{% endstep %} {% endstepper %} ### 🦙 Run Inkling in llama.cpp For this guide we'll be running the `UD-IQ1_S` quant which will require at least 290GB RAM. Feel free to change quantization type. For these tutorials, we will using [llama.cpp](llama.cpphttps://github.com/ggml-org/llama.cpp) for fast local inference. GGUF: [**Inkling-GGUF**](https://huggingface.co/unsloth/Inkling-GGUF) {% stepper %} {% step %} Obtain the SPECIFIC `llama.cpp` PR on [GitHub here](https://github.com/ggml-org/llama.cpp/pull/25731). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default. ```bash apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp gh pr checkout 25731 cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp ``` {% endstep %} {% step %} You can now use `llama.cpp` directly to load and download models, just like `ollama run`. First, select the quantization type you want like `Q2_K_XL`. Also use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Note this download process might be very slow, so it's probably best to use the manual download process in the next section. **Inkling-Small 276B:** ```bash export LLAMA_CACHE="unsloth/Inkling-Small-GGUF" ./build/bin/llama-cli \ -hf unsloth/Inkling-Small-GGUF:UD-Q3_K_XL \ --temp 1.0 \ --top-p 1.0 \ --min-p 0.0 ``` **Inkling 975B:** {% code overflow="wrap" %} ```bash export LLAMA_CACHE="unsloth/Inkling-GGUF" ./build/bin/llama-cli \ -hf unsloth/Inkling-GGUF:UD-IQ1_S \ --temp 1.0 \ --top-p 1.0 \ --min-p 0.0 ``` {% endcode %} {% endstep %} {% step %} If you want to download the model manually, we can download the model via the code below (after installing `pip install huggingface_hub`). If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging.md) **Inkling-Small 276B:** ```bash hf download unsloth/Inkling-Small-GGUF \ --local-dir unsloth/Inkling-Small-GGUF \ --include "*UD-Q3_K_XL*" # Use "*UD-Q4_K_XL*" for 4-bit ``` **Inkling 975B:** {% code overflow="wrap" %} ```bash hf download unsloth/Inkling-GGUF \ --local-dir unsloth/Inkling-GGUF \ --include "*UD-IQ1_S*" # Use "*UD-Q2_K_XL*" for 2-bit ``` {% endcode %} {% endstep %} {% step %} You can edit `--threads 32` for the number of CPU threads, `--ctx-size 32768` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. We implemented a Flash Attention bias fused kernel in llama.cpp as well to reduce VRAM usage - but best to keep `--ctx-size` modest still! **Inkling-Small 276B:** {% code overflow="wrap" %} ```bash ./build/bin/llama-cli \ --model unsloth/Inkling-Small-GGUF/UD-Q3_K_XL/Inkling-Small-UD-Q3_K_XL-00001-of-00004.gguf \ --temp 1.0 \ --top-p 1.0 \ --min-p 0.0 ``` {% endcode %} **Inkling 975B:** {% code overflow="wrap" %} ```bash ./build/bin/llama-cli \ --model unsloth/Inkling-GGUF/UD-IQ1_S/inkling-UD-IQ1_S-00001-of-00007.gguf \ --temp 1.0 \ --top-p 1.0 \ --min-p 0.0 ``` {% endcode %} {% endstep %} {% endstepper %} ### :triangular\_ruler:Quantization Analysis We also ran KLD (KL Divergence) benchmarks to gauge the accuracy of our quantizations of Inkling-GGUF. We first created a 8-bit + 6-bit quant mixture for the model with 6-bit for the ffn\_up and ffn\_gate exps, and found RMSE to be 1e-4 or less in terms of dequantization - quantizing ffn\_down increased error 10x more, so that is kept in Q8\_0 for the 8/6-bit quant On pure top-1% accuracy, **dynamic 1-bit gets around 74.2% accuracy yet being 86% smaller**! Dynamic 2-bit gets around 81% accuracy whilst being 82% smaller. This shows dynamically quantizing some layers to higher precision does not make the model unusable and fully useless - we show selectively quantizing layers can recover a lot of accuracy!
**But what does "74% accuracy" actually entail?** It does NOT mean 26% of the time the model outputs gibberish or wrong results - instead it means 26% of the top, it might choose the 2nd highest probable answer more. For example asking "Create a poem" will produce vastly different results - 74% of the time it'll be similar poems to the BF16 baseline, but 26% it'll be another poem, but still correct. For factual answers where creativity and sampling is not involved, the model will always recall the same answer (for example "What is 2+2", it'll always say 4, not 5 ever)
## 📊 Benchmarks You can view further below for Inkling benchmarks in table format: Inkling results are reported at effort=0.99. Comparison scores are generated Jul 14, 2026. Nemotron 3 Ultra, Kimi K2.5, Kimi K2.6, GLM 5.2, and DeepSeek V4 Pro are open weights models; Gemini 3.1 Pro, Claude Fable 5, and GPT 5.6 Sol are closed weights models. | | | Inkling | Nemotron 3 Ultra | Kimi K2.5 | Kimi K2.6 | GLM 5.2 | DeepSeek V4 Pro | Gemini 3.1 Pro (high) | Claude Fable 5 (max) | GPT 5.6 Sol (xhigh) | | --------------------- | --------------------------------- | ------- | ---------------- | --------- | --------- | ------- | --------------- | --------------------- | -------------------- | ------------------- | | **Reasoning** | | | | | | | | | | | | | HLE (text only) | 29.7% | 26.6% | 29.4% | 35.9% | 40.1% | 35.9% | 44.7% | 53.3% | 47.2% | | | HLE (with tools) | 46.0% | 37.4% | 50.2% | 54.0% | 54.7% | 48.2% | 51.4% | 64.5% | 55.0% | | | AIME 2026 | 97.1% | 94.2% | 95.8% | 96.4% | 99.2% | 96.7% | 98.3% | – | 99.9% | | | GPQA Diamond | 87.2% | 86.7% | 87.9% | 91.1% | 89.5% | 88.8% | 94.1% | 92.6% | 94.1% | | **Agentic (coding)** | | | | | | | | | | | | | SWEBench Verified | 77.6% | 70.7% | 76.8% | 80.2% | – | 80.6% | 80.6% | 95.0% | – | | | SWEBench Pro (Public) | 54.3% | 46.4% | 50.7% | 58.6% | 62.1% | 55.4% | 54.2% | 80.0% | 64.6% | | | Terminal Bench 2.1 (Best Harness) | 63.8 | 56.4 | 51.3 | 71.3 | 82.7 | 64 | 73.8 | 84.6 | 89.5 | | | GDPVal-AA v2 | 1233 | 1164 | 1009 | 1190 | 1514 | 1307 | 962 | 1760 | 1748 | | **Agentic (general)** | | | | | | | | | | | | | MCP Atlas | 74.1% | 44.7% | 64.0% | 68.1% | 77.8% | 73.2% | 78.2% | 83.3% | 81.8% | | | Tau 3 Banking | 23.7% | 13.8% | 13.2% | 20.6% | 26.8% | 25.8% | 16.5% | 26.8% | 33.0% | | **Factuality** | | | | | | | | | | | | | BrowseComp (w/ Ctx) | 77.1% | – | 74.9% | 83.2% | – | 83.4% | 85.9% | 88.0% | 89.4% | | | SimpleQA Verified | 43.9% | 32.4% | 36.9% | 38.7% | 38.1% | 57.0% | 77.3% | 68.3% | 71.6% | | | AA Omniscience | 1.0% | -1.0% | -8.0% | 6.0% | 4.0% | -10.0% | 33.0% | 40.0% | 22.0% | | **Chat** | | | | | | | | | | | | | IFBench | 79.8% | 81.4% | 70.2% | 76.0% | 73.3% | 76.5% | 77.1% | 63.5% | 72.7% | | | Global-MMLU-Lite | 88.7% | 85.6% | 84.0% | 88.4% | 89.2% | 89.3% | 92.7% | 93.3% | 91.8% | | **Vision** | | | | | | | | | | | | | MMMU Pro (Standard 10) | 73.3% | – | 75.0% | 79.0% | – | – | 82.0% | 84.2% | 83.0% | | | Charxiv RQ | 78.1% | – | 77.5% | 80.4% | – | – | 80.2% | 86.5% | 84.7% | | | Charxiv RQ (with python) | 82.0% | – | 78.7% | 86.7% | – | – | 89.9% | 89.4% | 87.8% | | **Audio** | | | | | | | | | | | | | Audio MC | 56.6% | – | – | – | – | – | 66.8% | – | – | | | MMAU | 77.2% | – | – | – | – | – | 82.5% | – | – | | | VoiceBench | 91.4% | – | – | – | – | – | 94.3% | – | – | | **Safety** | | | | | | | | | | | | | FORTRESS (Adversarial) | 78.0% | 77.6% | 54.1% | 65.6% | 71.3% | 36.0% | 65.2% | 96.0% | 82.4% | | | FORTRESS (Benign) | 95.9% | 90.5% | 98.3% | 97.2% | 90.0% | 98.5% | 98.0% | 55.1% | 98.1% | | | StrongREJECT | 98.6% | 98.7% | 99.5% | 99.8% | 98.5% | 98.6% | 98.0% | 98.7% | 98.5% | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://unsloth.ai/docs/models/inkling.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.