> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/basics/computer-use.md).

# Computer Use with Local LLMs

Let local models see your screen and use the mouse and keyboard in any app with Unsloth

Computer use lets a model see this computer's screen and control its mouse and keyboard in any app, the same way [Browser Use](/docs/basics/browser-use.md) drives the browser panel. It works in [Unsloth Desktop](/docs/desktop.md) on macOS, Windows and Linux, with local vision models (GGUF, safetensors, MLX) and with connected API providers.

{% columns %}
{% column width="50%" %}
In the example to the right, a local Qwen3.8-27B reads six receipt photos, builds a spreadsheet with a total, turns a company template into a three-slide summary, and flags a receipt photographed twice and one with no date.

Unsloth asks before a chat's first action, before each new app, and before anything risky. You can stop the model at any time by pressing **Stop** or moving the pointer into a corner of the screen.
{% endcolumn %}

{% column width="50%" %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsxPLKIGxOb9TvDdswlKA%2Fcomputer-use-receipts.gif?alt=media&amp;token=05e22940-83ce-484d-bb84-9e8151eab245" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### Quickstart

{% stepper %}
{% step %}

#### Download Unsloth

Install the Unsloth Desktop app for macOS, Windows or Linux.

<a href="https://unsloth.ai/download" class="button primary" data-icon="down-to-bracket">Download Unsloth</a>

Or install Unsloth natively:

{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

**Windows:**

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

{% endtab %}

{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}

{% tab title="Homebrew" %}
Install Unsloth Desktop:

{% code collapsedlinecount="10" %}

```bash
brew install --cask unsloth
```

{% endcode %}

Update Unsloth Desktop:

```bash
brew upgrade --cask unsloth
```

{% endtab %}

{% tab title="WinGet" %}
Install Unsloth Desktop:

```powershell
winget install --id UnslothAI.Unsloth -e
```

Update Unsloth Desktop:

```powershell
winget upgrade --id UnslothAI.Unsloth -e
```

{% endtab %}
{% endtabs %}

Install it directly on the computer you want the model to use. The Docker image and Unsloth inside WSL cannot see or control your desktop: Docker has no access to it, and WSL only sees its own Linux windows, not Windows.
{% endstep %}

{% step %}

#### Turn on computer use

Open Settings and select **Computer use**. Switch on **Computer use** and confirm. On macOS, allow **Screen & System Audio Recording** and **Accessibility** with the buttons on the same page, then restart Unsloth.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FcXTrPN6c1Tqe9hhMQTms%2Fcomputer-use-settings.png?alt=media" alt="" width="563"><figcaption></figcaption></figure></div>

Computer use is only offered when Unsloth serves the computer it runs on: a local launch, not a network bind, a Cloudflare link, Remote Access or a hosted notebook.
{% endstep %}

{% step %}

#### Choose a vision model

Computer use needs a model that can see images and call tools. These are the ones we ran on real desktop tasks:

| Model                                                                              | Notes                                                                                                                                               |
| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) (UD-Q4\_K\_XL) | Best in our tests: finished the receipts and GIMP tasks on its own. About 19 GB with its vision file                                                |
| [gemma-4-26B-A4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF)  | Fast, and clicked the right button in every pointing test, but in both receipts runs it kept reopening the photos and never reached the spreadsheet |
| [Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF)        | Fast mixture of experts; lost track of the receipts task in our tests                                                                               |

Small models such as gemma-4-E4B could not point at buttons at all.

Unsloth knows how each model points: Qwen3-VL, Qwen3.5 and later, Gemma 4, UI-TARS and Gemini answer on a 0 to 1000 grid, others in screenshot pixels. If clicks land in the wrong place with another model, change **How the model points** in Settings > Computer use.
{% endstep %}

{% step %}

#### Allow long tasks

A real task takes 100 to 200 actions, and a chat stops after 25 tool calls by default. In the chat settings panel, drag **Max Tool Calls Per Message** to **Max**.
{% endstep %}

{% step %}

#### Start a task

Open the **+** menu beside the chat input and enable **Computer use**. A **Computer** pill appears in the composer. Then describe the task, as in the example below.
{% endstep %}
{% endstepper %}

### Approvals and safety

Controlling the desktop is outside every sandbox, so computer use asks first:

* **The first action in each chat** asks, in every permission mode except Full access. The go-ahead lapses after 30 idle minutes.
* **Each new app** asks the first time a chat clicks or types in it. **Allow** covers that app for the chat; **Always allow \<app>** adds it to the list in Settings.
* **Risky steps** always ask, even in Approve for me: terminals, password managers, system tools, opening one of them, and shortcuts that close or delete.
* **Blocked apps** (Settings > Computer use > Blocked apps) are never clicked, typed into or opened, in any chat or permission mode.
* **Never done:** keys that lock the screen or end the session, input into an operating system security prompt, and input into Unsloth's own window.

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FlGFk1cVAfeUsugqUJXdk%2Fcomputer-use-approval.png?alt=media" alt="" width="563"><figcaption><p>From the receipts run: the file manager had not been used in this chat yet, so Unsloth asked first</p></figcaption></figure></div>

To take back control, press **Stop** in the chat, switch computer use off, or move the pointer into any corner of the screen. A corner stop holds until you answer the chat's next question or select **Allow again** in Settings.

Each tool card shows the steps the model asked for and the screenshot it got back, so you can see exactly what it saw.

### Example: an expense report from receipt photos

Six receipt photos are in `~/Documents/receipts`, and a company slide template is in `~/Documents`. One receipt was photographed twice and one has no date. No API reads a photo of a receipt, so this is work for computer use: the model looks at each picture, then fills in LibreOffice Calc and LibreOffice Impress.

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FsxPLKIGxOb9TvDdswlKA%2Fcomputer-use-receipts.gif?alt=media&amp;token=05e22940-83ce-484d-bb84-9e8151eab245" alt="" width="375"><figcaption></figcaption></figure>

{% code collapsedlinecount="10" %}

```
In ~/Documents/receipts there are 6 receipt photos. Open each one in
Image Viewer and read its merchant, date, category (Meals, Travel,
Office, Lodging or Software) and total; zoom in if the text is small.

Open LibreOffice Calc and enter one row per photo with the columns
File, Merchant, Date, Category and Total. Add a grand total with a SUM
formula. Then select the Merchant and Total columns (not the dates) and
insert a bar chart of total by merchant, placed to the right of the
table so it covers no cells. Save it as ~/Desktop/expenses.ods.

Then open ~/Documents/Northwind_template.pptx in LibreOffice Impress
and turn it into a 3-slide summary in the same style: a title slide, a
slide with the grand total and the three largest expenses, and a slide
listing anything that looks wrong, such as the same receipt
photographed twice or a receipt with no date. Check each slide for
text that overflows and fix it. Save it as ~/Desktop/expense_summary.odp
and export it to PDF on the Desktop.

Take a screenshot at the end, then tell me the grand total and what you
flagged.
```

{% endcode %}

Qwen3.8-27B read every receipt correctly, including the missing date, and flagged both problems on its own:

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FLmVbOlyIcEJEjBKBXbL7%2Fcomputer-use-receipts-result.png?alt=media" alt="" width="563"><figcaption><p>The model's answer at the end of the run</p></figcaption></figure></div>

It built the spreadsheet with a SUM total and a chart of the totals:

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Fl6Xp4xwydsvikJRjv1CN%2Fcomputer-use-receipts-calc.png?alt=media" alt="" width="563"><figcaption><p>The sheet and chart it made in LibreOffice Calc</p></figcaption></figure></div>

and turned the company template into three slides, flagging the duplicate and the missing date:

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2FpFJjGqRmEfpIJHwkFbCH%2Fcomputer-use-receipts-deck.png?alt=media" alt=""><figcaption><p>The exported deck</p></figcaption></figure></div>

We ran this task six times with Qwen3.8-27B on one A100, at about 130 tokens a second. Two runs got everything right, in 22 and 30 minutes; one of them even noted that the duplicate overstates the total by $585.08. Two misread one amount, one stopped before making the deck, and one charted the Date column. That last one used an earlier prompt: in the three runs with the prompt above, which names the columns, the chart was always right.

Check the result yourself, as you would a colleague's: a misread amount or text clipped at the edge of a slide is easy to miss.

### Example: a captioned image in GIMP

A shorter task in an app with its own menus, dialogs and tools. With GIMP already open, the model scales a photo, adds a caption and exports it:

{% code collapsedlinecount="10" %}

```
GIMP is open. Open ~/Pictures/photo.jpg (File > Open). Scale the image
to 1024 pixels wide with Image > Scale Image, keeping the aspect ratio.
Then use the Text tool to add the words "Northwind Labs" in white,
large enough to read, near the bottom left. Export the result as
~/Desktop/poster.png with File > Export As, accepting the export
dialog's defaults. Take a screenshot at the end.
```

{% endcode %}

<figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2F6ng8q3NylovIBoPFqZcm%2Fcomputer-use-gimp.gif?alt=media" alt=""><figcaption><p>Qwen3.8-27B (UD-Q4_K_XL) in GIMP: open, scale, set the text color, type the caption and export, 7 minutes shown 8x faster</p></figcaption></figure>

<div data-with-frame="true"><figure><img src="https://3215535692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FxhOjnexMCB3dmuQFQ2Zq%2Fuploads%2Flo1aum9JTcFJPfk37AyU%2Fcomputer-use-gimp-result.png?alt=media" alt="" width="563"><figcaption><p>The finished image in GIMP, scaled to 1024 by 576 with the caption on its own text layer</p></figcaption></figure></div>

Qwen3.8-27B did this right in both of our runs, in 7 and 11 minutes on one RTX PRO 6000: each exported a 1024 by 576 PNG with the white caption at the bottom left.

#### Tips for good results

* **Name the apps:** "open them in Image Viewer" is much more reliable than letting the model find an app.
* **Name exact columns, files and places:** "select the Merchant and Total columns", "save it as \~/Desktop/expenses.ods".
* **Say what done looks like:** the files to save, and what to tell you at the end.
* **Ask it to check:** "check each slide for text that overflows" or "take a screenshot at the end".
* **Give it room:** set **Max Tool Calls Per Message** to **Max**. A task like this takes 10 to 25 minutes on one GPU.
* **Prepare the screen:** open the files or folder yourself and close what the task does not need. Use a modest resolution on very large displays.
* **Watch the first minutes:** if the model heads for a terminal or the browser's developer tools to script its way around the screen, stop it and say "use only the app's own menus and buttons".

### Troubleshooting

<details>

<summary>Computer use is unavailable</summary>

Check Settings > Computer use for the reason. It needs a vision model that can call tools, the regular chat view, and Unsloth serving the machine it runs on. Only the installation owner can use it, and not through an API key. On Linux it needs an X11 session; Wayland sessions are not supported yet.

</details>

<details>

<summary>Clicks land in the wrong place</summary>

Change **How the model points** in Settings > Computer use between **Screenshot pixels** and **0 to 1000 grid**. With several displays, the model works on the main one unless it asks for a screenshot of another display by number.

</details>

<details>

<summary>The model stops after 25 steps</summary>

That is the default tool call limit for one reply. Drag **Max Tool Calls Per Message** to **Max** in the chat settings panel, then say "continue".

</details>

<details>

<summary>The model cannot open an app</summary>

Ask for the app by the name the launcher shows, such as "Image Viewer" or "LibreOffice Calc". On Linux, program names such as `eog` work too.

</details>

<details>

<summary>macOS says Unsloth cannot see the screen or use the keyboard</summary>

Open System Settings > Privacy & Security and allow Unsloth under **Screen & System Audio Recording** and **Accessibility**, then restart Unsloth. The buttons in Settings > Computer use open the right pages.

</details>

<details>

<summary>The model stopped and will not continue</summary>

If you moved the pointer into a corner, computer use stays stopped for that chat. Answer the chat's next approval, or select **Allow again** in Settings > Computer use.

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/basics/computer-use.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
