> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/get-started/install/amd/amd-npu.md).

# AMD NPU: Run Models Locally Guide

Run local AI on your AMD Ryzen AI NPU with Unsloth

You can now run local models on AMD Ryzen AI PCs with XDNA 2 NPUs by using the open-source package [Unsloth](https://github.com/unslothai/unsloth). NPUs deliver powerful compute performance while using 2–4× less energy per token generation than traditional GPUs. Enable the NPU runtime, download a supported model, and start chatting on Windows or Linux.

AMD Ryzen AI 300 (Strix Point, Krackan Point), Ryzen AI Max (Strix Halo) or Ryzen AI 400 (Gorgon Point), and future AMD devices with integrated NPUs are all supported! Read also our official [AMD NPU blog](https://www.amd.com/en/blogs/2026/unlocking-the-amd-ryzen-ai-npu-for-unsloth-desktop.html).

On supported PCs with an XDNA 2 NPU, compatible [Q4NX](https://github.com/ROCm/FLM_Q4NX_Converter) models appear in the model picker under NPU.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FMVl7QpH2e6o8Wq9TAziS%2F01-dots-v8.gif?alt=media&amp;token=10a47c25-2fb9-4843-a4a0-c589ede1d1ce" alt="" width="563"><figcaption></figcaption></figure>

#### Features

* **Run LLMs on the NPU**, with streaming responses and support for reasoning, vision and tools, skills, MCP servers, web search, chatting with local documents, and coding with local files.&#x20;
* Connect compatible apps and scripts to your NPU models with [**Unsloth's API endpoint**](https://unsloth.ai/docs/basics/api).
* **Offload Local inference to the NPU** and free up memory on your other processors.

#### Supported PCs

* **Processor:** AMD Ryzen AI 300 (Strix Point, Krackan Point), Ryzen AI Max (Strix Halo) or Ryzen AI 400 (Gorgon Point), with an XDNA 2 (or newer) NPU.
* **Operating system:** Windows 11 or Linux, x64.
* **Linux requirements:** kernel 7.0 or newer with the `amdxdna` driver, or an older kernel with `amdxdna-dkms`. You also need updated NPU firmware and an unlimited `memlock` setting.

### Getting Started

{% stepper %}
{% step %}

#### Install Unsloth

For this guide, use [**Unsloth Desktop**](https://unsloth.ai/docs/desktop) if you prefer a native app, or [**Unsloth Web UI**](https://unsloth.ai/docs/new/studio) if you prefer to work in your browser.

Follow the [**Unsloth install guide**](https://unsloth.ai/docs/get-started/install) to get started. On a supported Ryzen AI PC, the model picker automatically shows an **NPU** section.
{% endstep %}

{% step %}

#### Enable the NPU

Open the model picker, click the filter and change from All to NPU, then click **Enable NPU**. It's a one-time setup, Unsloth downloads [**FastFlowLM**](https://github.com/ROCm/FastFlowLM) and checks your NPU driver version.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FkVHZtnVPyZ9rrnKZdNmj%2Fimage.png?alt=media&amp;token=b0a1f65e-8ca6-45a2-864c-154acddc967f" alt="" width="563"><figcaption></figcaption></figure>

If something's missing, Unsloth tells you exactly what to fix: an old driver or firmware, a kernel without NPU support, or a memlock limit that's too low. It links you straight to the driver page.
{% endstep %}

{% step %}

#### Pick a model

Once setup finishes, the NPU models are listed. Filter by NPU and click one to download and load it.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F5gQAFd7G6TRBDguWPVhL%2Fimage.png?alt=media&amp;token=43d88f4f-70ef-45ee-a3cc-f0115e32e62a" alt="" width="297"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Start chatting

That's it, you can now load LLMs on the NPU. Set them up with access to the browser canvas, design skills to work on any task, and view the speed in tokens / sec.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fqt1DUsPrkl6VKKTQXmKY%2Fui-lattice-1440.gif?alt=media&amp;token=bed4b1b4-5dd2-499f-9250-f984217c1159" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Which models can I run?

You can also browse NPU models in the Model Hub by setting the format filter to NPU.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FGACGGPTdJFv2ZDIf0ewy%2Fimage.png?alt=media&amp;token=98227958-696c-4871-9a7a-95cd8a987c33" alt="" width="563"><figcaption></figcaption></figure>

The NPU runs Q4NX (4-bit) [models built for AMD's FastFlowLM runtime](https://huggingface.co/collections/FastFlowLM/flm-models-xdna2). Some highlights from the FastFlowLM models with their measured speed in **tokens per second** (tok/s) and **TTFT** (Time To First Token).

<table><thead><tr><th width="221">Model</th><th width="86">Size</th><th width="150">Capabilities</th><th width="98">Recommended RAM</th><th width="90">tok / s</th><th width="91">TTFT</th></tr></thead><tbody><tr><td>Qwen3.6-35B-A3B-NPU2</td><td>20.6 GB</td><td>Vision, reasoning, tools</td><td>64 GB</td><td>17.2</td><td>3.13 s</td></tr><tr><td>Qwen3.8-27B-NPU2 (MTP)</td><td>18.1 GB</td><td>Vision, reasoning, tools</td><td>48 GB</td><td>2.0</td><td>3.80 s</td></tr><tr><td>Gemma4-12B-IT-NPU2</td><td>8.7 GB</td><td>Vision, audio, reasoning, tools</td><td>32 GB</td><td>6.8</td><td>1.70 s</td></tr><tr><td>Qwen3.5-9B-NPU2</td><td>7.6 GB</td><td>Vision, reasoning, tools</td><td>32 GB</td><td>9.5</td><td>1.11 s</td></tr><tr><td>Qwen3-VL-4B-Instruct-NPU2</td><td>3.9 GB</td><td>Vision, tools</td><td>16 GB</td><td>20.5</td><td>1.05 s</td></tr><tr><td>LFM3-1.7B-NPU2</td><td>1.5 GB</td><td>Reasoning, tools</td><td>16 GB</td><td>40.7</td><td>0.12 s</td></tr></tbody></table>

FastFlowLM and the Q4NX converter are progressivesely adding new models, Unsloth will stay up to date with the latest versions of these repositories giving you access to the latest NPU models.

#### Why use the NPU?

Run a local chat model while keeping your GPU available for gaming, rendering or other GPU workloads. The NPU is a dedicated AI engine built into your Ryzen AI processor, giving you another way to run models on the hardware you already own.

For example, you can chat with a local model on the NPU while your Radeon GPU renders a video. In the model picker, choose a model from the NPU section. During chat, Unsloth shows its speed in tokens per second so you can see how it performs on your PC.

### Settings

Click the gear next to a downloaded NPU model to set its context length before loading. By default, Auto sizes the context to the RAM free when the model loads.

<div><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F7q0yRrzPivaPnuL9uecV%2Fimage.png?alt=media&amp;token=3ac95427-f5f7-40b0-90da-bf7c467666ba" alt="" width="375"><figcaption></figcaption></figure> <figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FUXrBPRzC4bqN6ZM9BuTv%2Fimage.png?alt=media&amp;token=94f1e046-9a37-4265-946c-d73eca9e141b" alt="" width="372"><figcaption></figcaption></figure></div>

**Power Mode** How hard the NPU works with modes `Power saver`, `Balanced`, `Performance` or `Turbo`. FastFlowLM’s default is `Performance`. Pick `Power saver` or `Balanced` when prioritizing battery powered setups, or `Turbo` for the fastest replies when you’re plugged in.

**Prefill Chunk Length** How many tokens of your prompt the NPU reads in one pass before it starts answering. It doesn’t limit how long a prompt can be (context length does that); a long prompt just takes more passes. Bigger chunks can get to the first token faster on long prompts. FastFlowLM’s default is 4096 for most models, and Unsloth keeps the chunk at or below the context length.

**Image Pre-resize** Shows up only for models that read images. It shrinks a picture before the model sees it, set to `Original size`, `480p`, `720p`, `1080p` or `1440p`. Smaller images are processed faster, and larger ones keep more detail.

Settings you change apply the next time the model loads, so click **Reload model** to use them right away. **Save, load** and **reset run settings** for a given mode&#x6C;**.**

#### **Use the NPU from the terminal and the API**

Serve a model with `unsloth run`. This starts a local server and loads Qwen3.6-35B-A3B-NPU2 on your NPU.

```bash
unsloth run --model Qwen3.6-35B-A3B-NPU2 --context-length 16384
```

To [connect an OpenAI-compatible chat app](https://unsloth.ai/docs/basics/api#connecting-your-client) or script, use the base URL and API key printed by `unsloth run`, with `Qwen3.6-35B-A3B-NPU2` as the model name.

If Unsloth is already running, use this command to start an interactive chat with Qwen3.6-35B-A3B-NPU2:

```bash
unsloth chat Qwen3.6-35B-A3B-NPU2
```

Here are a few flags you can use with `unsloth run` and `unsloth chat`. For the full list, run `unsloth run --help` or `unsloth chat --help`.

| Flag               | Command        | What it sets                                              |
| ------------------ | -------------- | --------------------------------------------------------- |
| `--model`          | `unsloth run`  | NPU model to load                                         |
| `--context-length` | `unsloth run`  | Total tokens shared by the prompt, chat history and reply |
| `--max-new-tokens` | `unsloth chat` | Maximum tokens per reply                                  |
| `--system-prompt`  | `unsloth chat` | Instructions for the model throughout the chat            |

NPU models serve through the OpenAI-compatible `/v1/chat/completions` endpoint. For embeddings, the Anthropic-compatible `/v1/messages` endpoint.

### Bring your own model

The NPU uses its own model format, called Q4NX. On the Export page you convert your GGUF models to Q4NX. The conversion itself runs on the CPU, so you can train on any GPU, export the GGUF, and convert it on your Ryzen AI PC. Then load your converted Q4NX model in a New Chat from the model picker under Trained.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fiezv9w2ejUOPXWHG0erv%2Fimage.png?alt=media&amp;token=99264d4c-9ef8-4ac8-be8a-c0f1ff65ab23" alt="" width="563"><figcaption></figcaption></figure>

#### Use the rest of your Ryzen AI PC

Your Radeon GPU handles supported training, image and video generation workloads. On Ryzen AI Max (Strix Halo), it can use a large share of system memory, allowing larger models to fit than on GPUs with less dedicated VRAM. Available memory depends on your PC’s configuration.

#### Secure by design

Unsloth manages its own copies of Lemonade and FastFlowLM, with separate runtime, configuration and model folders. Runtime downloads are pinned to specific versions and verified with SHA-256 checksums.

The managed Lemonade server accepts connections only from your PC and uses a new authentication key each time it starts. Unloading a model releases it from the NPU. Quitting Unsloth stops the managed runtime.

To hide NPU features and prevent the runtime from starting, set `UNSLOTH_STUDIO_ENABLE_NPU=0` before launching Unsloth.

#### FAQ

<details>

<summary>Does the NPU replace my GPU?</summary>

No. It’s a dedicated AI engine built into your Ryzen AI processor. Running a model on the NPU keeps GPU compute available for other workloads.

</details>

<details>

<summary>Can I run any GGUF on the NPU?</summary>

Not directly. This integration uses FastFlowLM’s Q4NX models. You can convert supported GGUF models to Q4NX, but loading your own conversions directly from the NPU model list is not yet supported.

</details>

<details>

<summary>My NPU isn't showing up. Why?</summary>

You need a supported AMD XDNA 2 NPU on Windows or Linux x64. If the NPU appears but setup fails, Unsloth checks your system and provides guidance for driver, firmware and other setup issues.

</details>

Thank you to AMD and the [Lemonade](https://github.com/lemonade-sdk/lemonade), [FastFlowLM](https://github.com/ROCm/FastFlowLM) and [Q4NX converter](https://github.com/ROCm/FLM_Q4NX_Converter) contributors for making this possible. AMD also wrote about this on their blog: <a href="https://www.amd.com/en/blogs/2026/unlocking-the-amd-ryzen-ai-npu-for-unsloth-desktop.html" target="_blank">Unlocking the AMD Ryzen AI NPU for Unsloth Desktop</a>.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/get-started/install/amd/amd-npu.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
