Inkling - How to Run Locally

Learn how to run Thinking Machine Labs' Inkling multimodal models locally.

Inkling by Thinking Machines Labs are open-weight multimodal models consisting of: Inkling-Small the new 276B (12B active) model and the earlier 975B (41B) parameter model. Licensed under Apache 2.0, the models support a 1M token context window, native multimodal input for text, image, and audio, and generates text as the output. Inkling excel at coding, agentic and tool-calling, RAG, chat, multilingual, and multimodal workloads.

Dynamic 2-bit reaches 81% top-1% accuracy whilst being 82% smaller. This shows if we shrink the model down by 82% with our Unsloth Dyanmic GGUF method - it does NOT mean the model gets 82% "dumber" - only ~18% degradation is seen. Thanks Thinking Machines Lab (TML) for giving Unsloth day-zero access. Inkling-Small-GGUF and Inkling-GGUF

Run Inkling Tutorials Quantization Results

⚙️ Usage Guide

The 1-bit dynamic quant UD-IQ1_S uses 270GB of disk space - this will need a Mac Studio Ultra or machines with at least RAM+VRAM to be 290GB or so.

The 1-bit quant will fit on a 290GB RAM and 6/8-bit requires 900GB RAM. See the 1-bit GGUF in action to the right where we asked Inkling to make a Sudoku HTML game.

See right for 1-bit Inkling GGUF in Unsloth:

Table: Inference hardware requirements (units = total memory: RAM + VRAM, or unified memory):

Model

2-bit

3-bit

4-bit

6/8bit

BF16

Inkling-Small

89 GB

128 GB

132 - 170 GB

256 GB

543 GB

Inkling

325 GB

450 GB

600 GB

870 GB

1900 GB

Table: Top-1% Retained (Accuracy Recovery) for 975B Inkling:

1-bit

2-bit

3-bit

4-bit

6/8bit

BF16

74.2%-77.4%

81.0%

88.7%

94.4%

99.8%

100.0%

For best performance, make sure your total available memory, including VRAM and system RAM, exceeds the quantized model file size by a comfortable margin.

Recommended Settings

Inkling has Non-thinking and Thinking modes. In Unsloth Studio you can easily toggle Thinking and non-Thinking with a UI.

Use these settings for most use cases:

Default Settings (Most Tasks)

temperature = 1.0

top_p = 1.0 (Off)

Maximum context window: 1,048,576.

Chat Template & thinking efforts

Inkling uses an interesting "Thinking effort level:" system prompt component where it's a number from 0.00 to 0.99! Tokenizing the below:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("unsloth/inkling")
tokenizer.apply_chat_template([
    {"role" : "user", "content" : "What is 1+1?"},
    {"role" : "assistant", "content" : "2!"},
    {"role" : "user", "content" : "What is 2+2?"},
], tokenize = False, reasoning_effort = "xhigh")

will get the below:

<|message_system|><|content_text|>Thinking effort level: 0.99<|end_message|><|message_user|><|content_text|>What is 1+1?<|end_message|><|message_model|><|content_text|>2!<|end_message|><|content_model_end_sampling|><|message_user|><|content_text|>What is 2+2?<|end_message|>

Disabling Thinking, changing reasoning effort

Inkling uses reasoning by default. It also supports reasoning efforts where reasoning_effort can be none = 0, low = 0.2, medium = 0.7, high = 0.9, xhigh = 0.99 and max = 0.99.

To disable thinking, use --chat-template-kwargs '{"reasoning_effort":'none'}'. If you're on Windows Powershell, use: --chat-template-kwargs "{\"reasoning_effort\":'none'}"

For reasoning effort customization and or to disable reasoning, use the below examples:

--chat-template-kwargs '{"reasoning_effort":"none"}'
--chat-template-kwargs '{"reasoning_effort":"low"}'
--chat-template-kwargs '{"reasoning_effort":"medium"}'
--chat-template-kwargs '{"reasoning_effort":"high"}'
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'
--chat-template-kwargs '{"reasoning_effort":"max"}'

Inkling also does interleaved thinking and tool calling - even with the dynamic 1-bit quant!

Run Inkling Tutorials:

Inkling runs in Unsloth on MacOS, Windows, and Linux. You can:

Search, download, run GGUFs and safetensor models
Self-healing tool calling + web search
Code execution (Python, Bash)
Automatic inference parameter tuning (temp, top-p, etc.)
Fast CPU + GPU inference via llama.cpp
Train LLMs 2x faster with 70% less VRAM

Install and Launch Unsloth

To install, run in your terminal:

MacOS, Linux, WSL:

curl -fsSL https://unsloth.ai/install.sh | sh

Windows PowerShell:

irm https://unsloth.ai/install.ps1 | iex

Launch Unsloth

MacOS, Linux, WSL and Windows:

Search and download Inkling

Then go to the Unsloth Chat tab and search for Inkling in the search bar and download your desired model and quant. Ensure you have enough compute the run the model.

Run Inkling

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our Unsloth Studio inference guide.

🦙 Run Inkling in llama.cpp

For this guide we'll be running the UD-IQ1_S quant which will require at least 290GB RAM. Feel free to change quantization type. For these tutorials, we will using llama.cpp for fast local inference. GGUF: Inkling-GGUF

Download Unsloth

Unsloth Guide

https://unsloth.ai/download

Obtain the SPECIFIC llama.cpp PR on GitHub here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default.

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
gh pr checkout 25731
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

You can now use llama.cpp directly to load and download models, just like ollama run. First, select the quantization type you want like Q2_K_XL. Also use export LLAMA_CACHE="folder" to force llama.cpp to save to a specific location. Note this download process might be very slow, so it's probably best to use the manual download process in the next section.

Inkling-Small 276B:

export LLAMA_CACHE="unsloth/Inkling-Small-GGUF"
./build/bin/llama-cli \
    -hf unsloth/Inkling-Small-GGUF:UD-Q3_K_XL \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0

Inkling 975B:

export LLAMA_CACHE="unsloth/Inkling-GGUF"
./build/bin/llama-cli \
    -hf unsloth/Inkling-GGUF:UD-IQ1_S \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0

If you want to download the model manually, we can download the model via the code below (after installing pip install huggingface_hub). If downloads get stuck, see: Hugging Face Hub, XET debugging

Inkling-Small 276B:

hf download unsloth/Inkling-Small-GGUF \
    --local-dir unsloth/Inkling-Small-GGUF \
    --include "*UD-Q3_K_XL*" # Use "*UD-Q4_K_XL*" for 4-bit

Inkling 975B:

hf download unsloth/Inkling-GGUF \
    --local-dir unsloth/Inkling-GGUF \
    --include "*UD-IQ1_S*" # Use "*UD-Q2_K_XL*" for 2-bit

You can edit --threads 32 for the number of CPU threads, --ctx-size 32768 for context length, --n-gpu-layers 2 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. We implemented a Flash Attention bias fused kernel in llama.cpp as well to reduce VRAM usage - but best to keep --ctx-size modest still!

Inkling-Small 276B:

./build/bin/llama-cli \
    --model unsloth/Inkling-Small-GGUF/UD-Q3_K_XL/Inkling-Small-UD-Q3_K_XL-00001-of-00004.gguf \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0

Inkling 975B:

./build/bin/llama-cli \
    --model unsloth/Inkling-GGUF/UD-IQ1_S/inkling-UD-IQ1_S-00001-of-00007.gguf \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0

📐Quantization Analysis

We also ran KLD (KL Divergence) benchmarks to gauge the accuracy of our quantizations of Inkling-GGUF. We first created a 8-bit + 6-bit quant mixture for the model with 6-bit for the ffn_up and ffn_gate exps, and found RMSE to be 1e-4 or less in terms of dequantization - quantizing ffn_down increased error 10x more, so that is kept in Q8_0 for the 8/6-bit quant

On pure top-1% accuracy, dynamic 1-bit gets around 74.2% accuracy yet being 86% smaller! Dynamic 2-bit gets around 81% accuracy whilst being 82% smaller. This shows dynamically quantizing some layers to higher precision does not make the model unusable and fully useless - we show selectively quantizing layers can recover a lot of accuracy!

But what does "74% accuracy" actually entail?

It does NOT mean 26% of the time the model outputs gibberish or wrong results - instead it means 26% of the top, it might choose the 2nd highest probable answer more.

For example asking "Create a poem" will produce vastly different results - 74% of the time it'll be similar poems to the BF16 baseline, but 26% it'll be another poem, but still correct. For factual answers where creativity and sampling is not involved, the model will always recall the same answer (for example "What is 2+2", it'll always say 4, not 5 ever)

📊 Benchmarks

You can view further below for Inkling benchmarks in table format:

Inkling results are reported at effort=0.99. Comparison scores are generated Jul 14, 2026. Nemotron 3 Ultra, Kimi K2.5, Kimi K2.6, GLM 5.2, and DeepSeek V4 Pro are open weights models; Gemini 3.1 Pro, Claude Fable 5, and GPT 5.6 Sol are closed weights models.

Inkling

Nemotron 3 Ultra

Kimi K2.5

Kimi K2.6

GLM 5.2

DeepSeek V4 Pro

Gemini 3.1 Pro (high)

Claude Fable 5 (max)

GPT 5.6 Sol (xhigh)

Reasoning

HLE (text only)

29.7%

26.6%

29.4%

35.9%

40.1%

35.9%

44.7%

53.3%

47.2%

HLE (with tools)

46.0%

37.4%

50.2%

54.0%

54.7%

48.2%

51.4%

64.5%

55.0%

AIME 2026

97.1%

94.2%

95.8%

96.4%

99.2%

96.7%

98.3%

–

99.9%

GPQA Diamond

87.2%

86.7%

87.9%

91.1%

89.5%

88.8%

94.1%

92.6%

94.1%

Agentic (coding)

SWEBench Verified

77.6%

70.7%

76.8%

80.2%

–

80.6%

95.0%

–

SWEBench Pro (Public)

54.3%

46.4%

50.7%

58.6%

62.1%

55.4%

54.2%

80.0%

64.6%

Terminal Bench 2.1 (Best Harness)

63.8

56.4

51.3

71.3

82.7

73.8

84.6

89.5

GDPVal-AA v2

1233

1164

1009

1190

1514

1307

962

1760

1748

Agentic (general)

MCP Atlas

74.1%

44.7%

64.0%

68.1%

77.8%

73.2%

78.2%

83.3%

81.8%

Tau 3 Banking

23.7%

13.8%

13.2%

20.6%

26.8%

25.8%

16.5%

26.8%

33.0%

Factuality

BrowseComp (w/ Ctx)

77.1%

–

74.9%

83.2%

–

83.4%

85.9%

88.0%

89.4%

SimpleQA Verified

43.9%

32.4%

36.9%

38.7%

38.1%

57.0%

77.3%

68.3%

71.6%

AA Omniscience

1.0%

-1.0%

-8.0%

6.0%

4.0%

-10.0%

33.0%

40.0%

22.0%

Chat

IFBench

79.8%

81.4%

70.2%

76.0%

73.3%

76.5%

77.1%

63.5%

72.7%

Global-MMLU-Lite

88.7%

85.6%

84.0%

88.4%

89.2%

89.3%

92.7%

93.3%

91.8%

Vision

MMMU Pro (Standard 10)

73.3%

–

75.0%

79.0%

–

82.0%

84.2%

83.0%

Charxiv RQ

78.1%

–

77.5%

80.4%

–

80.2%

86.5%

84.7%

Charxiv RQ (with python)

82.0%

–

78.7%

86.7%

–

89.9%

89.4%

87.8%

Audio

Audio MC

56.6%

–

66.8%

–

MMAU

77.2%

–

82.5%

–

VoiceBench

91.4%

–

94.3%

–

Safety

FORTRESS (Adversarial)

78.0%

77.6%

54.1%

65.6%

71.3%

36.0%

65.2%

96.0%

82.4%

FORTRESS (Benign)

95.9%

90.5%

98.3%

97.2%

90.0%

98.5%

98.0%

55.1%

98.1%

StrongREJECT

98.6%

98.7%

99.5%

99.8%

98.5%

98.6%

98.0%

98.7%

98.5%

Last updated 4 days ago

Was this helpful?