> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/models/inkling-2.md).

# Inkling 2 Small - Run Locally

Learn how to run Thinking Machines Lab’s Inkling 2 Small locally with Unsloth and llama.cpp.

**Inkling 2 Small** by [Thinking Machines Lab](https://thinkingmachines.ai/news/inkling-2-small/) is a **594B** mixture-of-experts model with **17B active parameters**. It supports text and image inputs, text output a **1M-token context window** and this guide will show you how you can run locally with GGUFs. Inkling 2 Small scores 72.1% on DeepSWE v1.1 and 86.3% on SWEBench Pro v2 (Hard). Thank you Thinking Machines for day zero access.

Run locally with Unsloth Dynamic GGUFs: [**Inkling-2-Small-GGUF**](https://huggingface.co/unsloth/Inkling-2-Small-GGUF). UD-IQ1\_S (dynamic 1bit) needs 171.7GB vs 394GB (NVFP4) and retains 89.7% of top-1% accuracy and is 57% smaller.

<a href="https://unsloth.ai/docs/models/inkling-2#run-inkling-2-small-tutorials" class="button primary">Run Inkling 2 Small Tutorials</a><a href="https://unsloth.ai/docs/models/inkling-2#quantization-analysis" class="button secondary">Quantization Results</a>

#### ⚙️ Usage Guide

The smallest quant, `UD-IQ1_S`, needs **171.7 GB of disk space for weights alone**. The examples below use the 2-bit `UD-IQ2_XXS` (**190.6 GB**). Choose a quant that fits your available **RAM + VRAM**, or unified memory.

{% hint style="success" %}
Leave additional memory for the operating system, KV cache, runtime buffers and vision projector. File size is not the total inference memory requirement. Start with a 32K context; longer contexts need more memory.
{% endhint %}

**Recommended Settings**

Use these starting settings. Thinking Machines evaluates Inkling 2 Small at **temperature 1.0** and **xhigh** reasoning effort.

| Temperature | Top-p | Top-k | Min-p |
| ----------- | ----- | ----- | ----- |
| `1.0`       | `1.0` | `0`   | `0.0` |

* **Maximum context window:** `1,048,576` tokens.
* Use the **embedded chat template**. In llama.cpp, enable it with `--jinja`.

**Changing reasoning effort**

The GGUF template defaults to **high**. Choose `low` for shorter reasoning, `xhigh` for difficult tasks, or `none` to request non-thinking responses. The `none` setting does not strictly guarantee zero reasoning.

| Effort | none | minimal | low | medium | high | xhigh / max |
| ------ | ---- | ------- | --- | ------ | ---- | ----------- |
| Value  | 0.0  | 0.1     | 0.2 | 0.7    | 0.9  | 0.99        |

### Run Inkling 2 Small Tutorials:

Use an **Unsloth / llama.cpp runtime with Inkling 2 support**. Older builds that support the original Inkling may not recognize the new `inkling2` architecture.

#### 🦥 Run Inkling 2 Small in Unsloth

{% stepper %}
{% step %}
**Install and launch Unsloth**

Download [**Unsloth Desktop**](https://unsloth.ai/download) for **macOS, Windows or Linux**. Or install from your terminal:

**macOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows PowerShell:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endstep %}

{% step %}
**Search and download Inkling 2 Small**

Open **Chat** or **Model hub**, search for **Inkling-2-Small** and select [**unsloth/Inkling-2-Small-GGUF**](https://huggingface.co/unsloth/Inkling-2-Small-GGUF). Download your chosen quant. For image inputs, also use the matching `mmproj-F16.gguf` vision projector (**658 MB**).
{% endstep %}

{% step %}
**Run Inkling 2 Small**

Load the model, start with **32,768 context tokens**, and use the settings above. You can adjust context length, sampling and reasoning effort in the model settings. For more information, see our [Unsloth inference guide](https://unsloth.ai/docs/new/studio/chat).

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsZTREyAjLXIj4ELXLDsz%2Fimage.png?alt=media&amp;token=130fd6b0-941e-4784-bf96-59220e55c743" alt=""><figcaption></figcaption></figure>

We prompted it to do a simple Flappy Bird game using UD-IQ1\_S and it worked well! We also provide the code it generated below

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FMV608A1O4RDQ25uBAko7%2Fimage.png?alt=media&amp;token=059b77ec-f05f-4642-9e53-b99accdc2243" alt="" width="188"><figcaption></figcaption></figure>

{% code overflow="wrap" expandable="true" %}

```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">
<title>Flappy Bird</title>
<style>
  * { margin: 0; padding: 0; box-sizing: border-box; -webkit-tap-highlight-color: transparent; }
  html, body {
    height: 100%;
    overflow: hidden;
    background: linear-gradient(#1a2a6c, #203a43, #2c5364);
    display: flex;
    justify-content: center;
    align-items: center;
    font-family: 'Courier New', monospace;
  }
  #wrap {
    position: relative;
    box-shadow: 0 0 60px rgba(0,0,0,.6), 0 0 0 6px #3a2a1a, 0 0 0 12px #57422f;
    border-radius: 4px;
    overflow: hidden;
  }
  canvas {
    display: block;
    background: #70c5ce;
    cursor: pointer;
  }
  #hint {
    position: absolute;
    bottom: 8px;
    left: 0; right: 0;
    text-align: center;
    color: rgba(255,255,255,.85);
    font-size: 12px;
    pointer-events: none;
    text-shadow: 1px 1px 2px #000;
  }
</style>
</head>
<body>
<div id="wrap">
  <canvas id="game" width="400" height="560"></canvas>
  <div id="hint">SPACE / CLICK / TAP to flap &nbsp;•&nbsp; P = pause</div>
</div>

<script>
(() => {
  const canvas = document.getElementById('game');
  const ctx = canvas.getContext('2d');
  const W = canvas.width, H = canvas.height;

  // ---------- Game constants ----------
  const GRAVITY = 0.38;
  const FLAP = -7.2;
  const PIPE_W = 62;
  const PIPE_GAP = 145;
  const PIPE_SPEED = 2.6;
  const PIPE_SPAWN = 1550;      // ms between pipes
  const GROUND_H = 72;
  const BIRD_X = 105;
  const BIRD_R = 14;

  // ---------- State ----------
  const STATE = { MENU: 0, PLAY: 1, OVER: 2, PAUSED: 3 };
  let state = STATE.MENU;
  let bird, pipes, score, best = 0, frame, lastPipe, groundX, flash;
  let clouds = [];

  try { best = parseInt(localStorage.getItem('flappy_best') || '0', 10) || 0; } catch (e) { best = 0; }

  function reset() {
    bird = { y: H / 2 - 40, vy: 0, rot: 0, wing: 0 };
    pipes = [];
    score = 0;
    frame = 0;
    lastPipe = 0;
    groundX = 0;
    flash = 0;
    clouds = [];
    for (let i = 0; i < 5; i++) {
      clouds.push({ x: Math.random() * W, y: 30 + Math.random() * 150, s: 0.6 + Math.random() * 0.8, v: 0.3 + Math.random() * 0.3 });
    }
  }
  reset();

  // ---------- Input ----------
  function flap() {
    if (state === STATE.MENU) { state = STATE.PLAY; bird.vy = FLAP; return; }
    if (state === STATE.OVER) { reset(); state = STATE.MENU; return; }
    if (state === STATE.PAUSED) { state = STATE.PLAY; return; }
    bird.vy = FLAP;
  }
  window.addEventListener('keydown', e => {
    if (e.code === 'Space' || e.code === 'ArrowUp') { e.preventDefault(); flap(); }
    if (e.code === 'KeyP') {
      if (state === STATE.PLAY) state = STATE.PAUSED;
      else if (state === STATE.PAUSED) state = STATE.PLAY;
    }
  });
  canvas.addEventListener('mousedown', flap);
  canvas.addEventListener('touchstart', e => { e.preventDefault(); flap(); }, { passive: false });

  // ---------- Update ----------
  function update(dt) {
    frame++;
    groundX = (groundX - PIPE_SPEED) % 48;
    clouds.forEach(c => { c.x -= c.v; if (c.x < -80) { c.x = W + 40; c.y = 30 + Math.random() * 150; } });

    if (state !== STATE.PLAY) {
      if (state === STATE.MENU) {
        bird.y = H / 2 - 40 + Math.sin(frame / 22) * 10;
        bird.wing = Math.sin(frame / 6) * 0.6;
      }
      if (flash > 0) flash -= dt;
      return;
    }

    bird.vy += GRAVITY;
    bird.y += bird.vy;
    bird.rot = Math.max(-0.5, Math.min(1.35, bird.vy / 9));
    bird.wing = Math.sin(frame / 5) * 0.7;

    // Spawn pipes
    const now = performance.now();
    if (now - lastPipe > PIPE_SPAWN) {
      lastPipe = now;
      const gapY = 90 + Math.random() * (H - GROUND_H - 180);
      pipes.push({ x: W + 20, gapY, passed: false });
    }

    // Move & score pipes
    for (const p of pipes) {
      p.x -= PIPE_SPEED;
      if (!p.passed && p.x + PIPE_W < BIRD_X) { p.passed = true; score++; }
    }
    pipes = pipes.filter(p => p.x + PIPE_W > -10);

    // Collisions
    const groundY = H - GROUND_H;
    if (bird.y + BIRD_R >= groundY) { bird.y = groundY - BIRD_R; die(); }
    if (bird.y - BIRD_R <= 0) { bird.y = BIRD_R; bird.vy = 0; }
    for (const p of pipes) {
      if (BIRD_X + BIRD_R > p.x && BIRD_X - BIRD_R < p.x + PIPE_W) {
        if (bird.y - BIRD_R < p.gapY || bird.y + BIRD_R > p.gapY + PIPE_GAP) { die(); break; }
      }
    }
    if (flash > 0) flash -= dt;
  }

  function die() {
    if (state !== STATE.PLAY) return;
    state = STATE.OVER;
    flash = 220;
    if (score > best) {
      best = score;
      try { localStorage.setItem('flappy_best', String(best)); } catch (e) {}
    }
  }

  // ---------- Drawing helpers ----------
  function drawBackground() {
    // Sky gradient
    const g = ctx.createLinearGradient(0, 0, 0, H - GROUND_H);
    g.addColorStop(0, '#4ec0ca');
    g.addColorStop(0.7, '#71d3dc');
    g.addColorStop(1, '#a8e6ec');
    ctx.fillStyle = g;
    ctx.fillRect(0, 0, W, H - GROUND_H);

    // Sun
    ctx.fillStyle = '#fff8c9';
    ctx.beginPath(); ctx.arc(W - 70, 62, 30, 0, Math.PI * 2); ctx.fill();
    ctx.fillStyle = 'rgba(255,248,201,.35)';
    ctx.beginPath(); ctx.arc(W - 70, 62, 46, 0, Math.PI * 2); ctx.fill();

    // Clouds
    ctx.fillStyle = 'rgba(255,255,255,.9)';
    for (const c of clouds) {
      ctx.save(); ctx.translate(c.x, c.y); ctx.scale(c.s, c.s);
      ctx.beginPath();
      ctx.arc(0, 0, 14, 0, Math.PI * 2);
      ctx.arc(14, -6, 12, 0, Math.PI * 2);
      ctx.arc(28, 0, 14, 0, Math.PI * 2);
      ctx.arc(14, 5, 15, 0, Math.PI * 2);
      ctx.fill();
      ctx.restore();
    }

    // Distant hills
    ctx.fillStyle = '#5dbb63';
    ctx.beginPath();
    ctx.moveTo(0, H - GROUND_H);
    for (let x = 0; x <= W; x += 40) {
      ctx.lineTo(x, H - GROUND_H - 26 - Math.sin((x + 200) / 55) * 14);
    }
    ctx.lineTo(W, H - GROUND_H);
    ctx.closePath(); ctx.fill();

    // Near bushes
    ctx.fillStyle = '#3fa34d';
    ctx.fillRect(0, H - GROUND_H - 6, W, 8);
  }

  function drawPipe(p) {
    const topH = p.gapY, botY = p.gapY + PIPE_GAP, botH = H - GROUND_H - botY;
    const grad = ctx.createLinearGradient(p.x, 0, p.x + PIPE_W, 0);
    grad.addColorStop(0, '#5aa832');
    grad.addColorStop(0.18, '#8ee06a');
    grad.addColorStop(0.35, '#73c74f');
    grad.addColorStop(0.55, '#4f9c2c');
    grad.addColorStop(0.8, '#3d7d22');
    grad.addColorStop(1, '#2c5c17');

    function pipeBody(y, h, capAtBottom) {
      ctx.fillStyle = grad;
      ctx.fillRect(p.x, y, PIPE_W, h);
      ctx.strokeStyle = '#1e4010'; ctx.lineWidth = 2;
      ctx.strokeRect(p.x, y, PIPE_W, h);
      // Cap
      const capH = 24, capY = capAtBottom ? y + h - capH : y;
      ctx.fillStyle = grad;
      ctx.fillRect(p.x - 4, capY, PIPE_W + 8, capH);
      ctx.strokeRect(p.x - 4, capY, PIPE_W + 8, capH);
    }
    pipeBody(0, topH, true);
    pipeBody(botY, botH, false);
  }

  function drawGround() {
    const y = H - GROUND_H;
    // Grass
    ctx.fillStyle = '#ded895';
    ctx.fillRect(0, y, W, GROUND_H);
    ctx.fillStyle = '#7ee35b';
    ctx.fillRect(0, y, W, 16);
    ctx.fillStyle = '#4cbb2e';
    for (let x = groundX; x < W + 48; x += 48) {
      ctx.beginPath();
      ctx.moveTo(x, y + 16); ctx.lineTo(x + 12, y + 2); ctx.lineTo(x + 24, y + 16);
      ctx.fill();
    }
    // Dirt stripes
    ctx.fillStyle = '#c9bd7a';
    for (let x = groundX; x < W + 48; x += 48) {
      ctx.fillRect(x + 6, y + 34, 22, 6);
      ctx.fillRect(x + 26, y + 52, 18, 6);
    }
    ctx.strokeStyle = '#3a2a1a'; ctx.lineWidth = 3;
    ctx.beginPath(); ctx.moveTo(0, y + 1.5); ctx.lineTo(W, y + 1.5); ctx.stroke();
  }

  function drawBird() {
    ctx.save();
    ctx.translate(BIRD_X, bird.y);
    ctx.rotate(bird.rot);

    // Body
    const bg = ctx.createRadialGradient(-4, -4, 2, 0, 0, BIRD_R + 2);
    bg.addColorStop(0, '#ffe873');
    bg.addColorStop(0.6, '#ffd23f');
    bg.addColorStop(1, '#e0a92e');
    ctx.fillStyle = bg;
    ctx.beginPath(); ctx.ellipse(0, 0, BIRD_R + 2, BIRD_R, 0, 0, Math.PI * 2); ctx.fill();
    ctx.strokeStyle = '#7a5b12'; ctx.lineWidth = 2; ctx.stroke();

    // Wing
    ctx.save();
    ctx.translate(-4, 0);
    ctx.rotate(bird.wing);
    ctx.fillStyle = '#f4b63f';
    ctx.beginPath(); ctx.ellipse(-4, 2, 9, 6.5, 0, 0, Math.PI * 2); ctx.fill();
    ctx.strokeStyle = '#7a5b12'; ctx.stroke();
    ctx.restore();

    // Eye
    ctx.fillStyle = '#fff';
    ctx.beginPath(); ctx.arc(7, -4, 5.5, 0, Math.PI * 2); ctx.fill();
    ctx.strokeStyle = '#333'; ctx.lineWidth = 1.2; ctx.stroke();
    ctx.fillStyle = '#222';
    ctx.beginPath(); ctx.arc(8.5, -4, 2.6, 0, Math.PI * 2); ctx.fill();

    // Beak
    ctx.fillStyle = '#ff6b35';
    ctx.beginPath();
    ctx.moveTo(BIRD_R - 1, -1);
    ctx.lineTo(BIRD_R + 9, 2.5);
    ctx.lineTo(BIRD_R - 1, 6);
    ctx.closePath(); ctx.fill();
    ctx.strokeStyle = '#a83a12'; ctx.lineWidth = 1.5; ctx.stroke();

    ctx.restore();
  }

  function drawText(text, x, y, size, color = '#fff', stroke = '#000') {
    ctx.font = `bold ${size}px 'Courier New', monospace`;
    ctx.textAlign = 'center';
    ctx.lineWidth = Math.max(2, size / 8);
    ctx.strokeStyle = stroke;
    ctx.strokeText(text, x, y);
    ctx.fillStyle = color;
    ctx.fillText(text, x, y);
  }

  function drawMedal(cx, cy, s) {
    const colors = ['#cd7f32', '#c0c0c0', '#ffd700', '#b9f2ff'];
    const idx = score >= 40 ? 3 : score >= 30 ? 2 : score >= 20 ? 1 : 0;
    ctx.fillStyle = colors[idx];
    ctx.beginPath(); ctx.arc(cx, cy, 20 * s, 0, Math.PI * 2); ctx.fill();
    ctx.strokeStyle = '#5a4a1a'; ctx.lineWidth = 2; ctx.stroke();
    ctx.fillStyle = 'rgba(255,255,255,.5)';
    ctx.beginPath(); ctx.arc(cx - 5 * s, cy - 5 * s, 7 * s, 0, Math.PI * 2); ctx.fill();
    drawText(['BRONZE','SILVER','GOLD','PLATINUM'][idx], cx, cy + 34 * s, 10, '#fff');
  }

  function draw() {
    ctx.clearRect(0, 0, W, H);
    drawBackground();
    pipes.forEach(drawPipe);
    drawGround();
    drawBird();

    if (state === STATE.PLAY || state === STATE.PAUSED) {
      drawText(String(score), W / 2, 78, 52);
    }

    if (state === STATE.MENU) {
      drawText('FLAPPY', W / 2, 130, 56, '#ffd23f', '#5a3a00');
      drawText('BIRD', W / 2, 190, 56, '#ffd23f', '#5a3a00');
      drawText('Tap / Space to flap', W / 2, 300, 17);
      drawText('Best: ' + best, W / 2, 330, 15, '#fff');
    }

    if (state === STATE.PAUSED) {
      ctx.fillStyle = 'rgba(0,0,0,.45)'; ctx.fillRect(0, 0, W, H);
      drawText('PAUSED', W / 2, H / 2 - 10, 40);
      drawText('Press P to resume', W / 2, H / 2 + 28, 16);
    }

    if (state === STATE.OVER) {
      if (flash > 0) { ctx.fillStyle = `rgba(255,255,255,${flash / 220 * 0.8})`; ctx.fillRect(0, 0, W, H); }
      // Game over panel
      ctx.fillStyle = 'rgba(0,0,0,.35)'; ctx.fillRect(0, 0, W, H);
      ctx.fillStyle = '#f5deb3';
      ctx.strokeStyle = '#5a3a1a'; ctx.lineWidth = 4;
      const px = W / 2 - 130, py = 150, pw = 260, ph = 190;
      ctx.beginPath(); ctx.roundRect(px, py, pw, ph, 10); ctx.fill(); ctx.stroke();
      drawText('GAME OVER', W / 2, py + 38, 30, '#e74c3c', '#3a1000');
      drawText('Score: ' + score, px + 130, py + 78, 20, '#333', 'rgba(0,0,0,0)');
      drawText('Best:  ' + best, px + 130, py + 104, 20, '#333', 'rgba(0,0,0,0)');
      drawMedal(px + 55, py + 140, 1);
      drawText('Tap to retry', px + 165, py + 148, 16, '#2c5c17', 'rgba(0,0,0,0)');
    }
  }

  // ---------- Main loop ----------
  let last = performance.now();
  function loop(now) {
    const dt = Math.min(50, now - last);
    last = now;
    update(dt);
    draw();
    requestAnimationFrame(loop);
  }
  requestAnimationFrame(loop);
})();
</script>
</body>
</html>

```

{% endcode %}
{% endstep %}
{% endstepper %}

#### 🦙 Run Inkling 2 Small in llama.cpp

Install [llama.cpp](https://github.com/ggml-org/llama.cpp) with Inkling 2 support - use the upstream [PR 32187](https://github.com/ggml-org/llama.cpp/pull/32187). Run the following commands from your llama.cpp directory using Bash (macOS, Linux or WSL).

On Ubuntu / WSL, install the build dependencies:

```bash
sudo apt-get update
sudo apt-get install -y build-essential cmake git libssl-dev
```

For NVIDIA, install the [CUDA toolkit](https://developer.nvidia.com/cuda-downloads). On macOS, install Xcode command line tools and CMake; use `-DGGML_CUDA=OFF` (Metal is enabled by default). Use `OFF` for CPU-only builds too.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev git -y
git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp fetch origin pull/32187/head:inkling2
git -C llama.cpp checkout inkling2
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

**1. Download the model and vision projector**

```bash
pip install -U huggingface_hub
hf download unsloth/Inkling-2-Small-GGUF \
    --include "UD-IQ2_XXS/*" "mmproj-F16.gguf" \
    --local-dir Inkling-2-Small-GGUF
```

Keep **all five shards** in the same folder. Pass the first shard to llama.cpp; the remaining shards load automatically. While the repository is private, downloads require access and `hf auth login`.

**2. Launch the local chat server**

{% code overflow="wrap" %}

```bash
./llama.cpp/llama-server \
    -m Inkling-2-Small-GGUF/UD-IQ2_XXS/Inkling-2-Small-UD-IQ2_XXS-00001-of-00005.gguf \
    --mmproj Inkling-2-Small-GGUF/mmproj-F16.gguf \
    --jinja -ngl 99 -c 32768 \
    --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
    --chat-template-kwargs '{"reasoning_effort":"high"}'
```

{% endcode %}

Open [**http://localhost:8080**](http://localhost:8080) to chat and upload images. Lower `-ngl` if the model does not fit in VRAM, or use `-ngl 0` for CPU inference. For text-only use, omit `--mmproj`.

#### Quantization Analysis

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fa3FEbHeane8hoZhRLuV5%2Fimage.png?alt=media&amp;token=84c3f9ed-cdcb-46d1-b715-283d47499ce7" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FHY57QxZ6VDo8haLPyusw%2Fimage.png?alt=media&amp;token=504b0604-944c-47f4-a725-b1883c1c9464" alt=""><figcaption></figcaption></figure>

Results from the [Unsloth GGUF model card](https://huggingface.co/unsloth/Inkling-2-Small-GGUF) compare each quant with the reference GGUF on **64 × 4,096 tokens of WikiText-2**, using F16 KV cache. **Top-token agreement is not a task benchmark accuracy score.** Sizes exclude the vision projector.

| Quant        | Size (GB) | Mean KLD ↓ | Same top token ↑ |
| ------------ | --------- | ---------- | ---------------- |
| `UD-IQ1_S`   | 171.7     | 0.0922     | 89.7%            |
| `UD-IQ1_M`   | 178.5     | 0.0829     | 90.2%            |
| `UD-IQ2_XXS` | 190.6     | 0.0577     | 91.8%            |
| `UD-Q2_K_XL` | 205.0     | 0.0450     | 92.7%            |
| `UD-IQ3_XXS` | 226.0     | 0.0364     | 93.3%            |
| `UD-Q3_K_XL` | 285.1     | 0.0176     | 95.3%            |
| `UD-Q4_K_XL` | 354.2     | 0.0029     | 98.0%            |
| `UD-Q8_K_XL` | 386.5     | 0.0027     | 98.1%            |
| `BF16`       | 484.4     |            |                  |

**BF16** is a lossless conversion that preserves the original **NVFP4 experts in layers 0–39** and stores the other weights in BF16. **UD-Q8\_K\_XL** preserves those same NVFP4 experts and quantizes the other weights to 8-bit.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/models/inkling-2.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.