> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/integrations/hermes-agent.md).
# How to Run Local AI Models with Hermes Agent
This guide enables you to run open LLMs locally with **Hermes Agent** via [**Unsloth**](https://github.com/unslothai/unsloth). Hermes Agent is an **open-source** autonomous AI agent that connects to a model endpoint, executes tasks, and improves over time through memory and learned skills.
It works with any **local model** exposed through Unsloth’s **OpenAI-compatible API**, including: DeepSeek, Qwen, Gemma, and more. Hermes acts as the agent client, while Unsloth loads and serves models via a local API.
After setup, every prompt sent through Hermes will run against your local model instead of a remote provider.
Setup Hermes🦥 Use open models with Unsloth
{% hint style="info" %}
In this tutorial, you’ll install Hermes and configure it to use `unsloth/Qwen3.6-27B-GGUF` served from Unsloth. Prefer a different model? Swap in any other model by loading it in Unsloth and updating the configuration.
{% endhint %}
### :caduceus: Setup Hermes Agent
**Prerequisites.** The installer verifies these and halts if any are missing. Install what's not already on your machine first:
* **OS** Linux, macOS, or Windows via WSL.
* **uv** Python package manager. Install with `curl -LsSf https://astral.sh/uv/install.sh | sh`.
* **Python 3.11+** the installer can provision this via `uv` if it's missing.
* **Git** to clone the Hermes repo.
* **Node.js** 18+ for Hermes's browser tools.
* **ripgrep** (`rg`) for fast file search.
* **ffmpeg** for TTS/voice messages.
#### **1. Run the installer** in a terminal:
```bash
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
```
The installer will:
1. Detect your OS.
2. Verify every prerequisite listed above and print a ✓ or ✗ for each.
3. Clone Hermes into `~/.hermes/hermes-agent/` (over SSH if a GitHub SSH key is configured, otherwise HTTPS).
4. Create a Python 3.11 virtualenv at `~/.hermes/hermes-agent/venv/`.
5. Install Hermes and all Python dependencies.
6. Install Node.js dependencies for the browser tools.
7. Install Playwright's Chromium engine. **This step prompts for `sudo`** so Playwright can install shared libraries. Hermes itself does not require root.
#### **2. Reload your shell** so the `hermes` command is on your `PATH`:
{% code title="bash" %}
```bash
source ~/.bashrc
```
{% endcode %}
{% code title="zsh" %}
```bash
source ~/.zshrc
```
{% endcode %}
#### **3. Verify the install:**
```bash
hermes --version
```
If the command resolves, Hermes is installed. Everything lives under `~/.hermes/`:
| Path | What it is |
| --------------------------------------- | ---------------------------------------------- |
| `~/.hermes/config.yaml` | Main settings (model, provider, tools, TTS, …) |
| `~/.hermes/.env` | API keys and other secrets |
| `~/.hermes/hermes-agent/` | The Hermes source + virtualenv |
| `~/.hermes/cron/`, `sessions/`, `logs/` | Runtime data |
| `~/.hermes/skills/` | Installed skills (synced from the Skills Hub) |
{% hint style="info" %}
Full install reference: [hermes-agent.nousresearch.com/docs/getting-started/installation](https://hermes-agent.nousresearch.com/docs/getting-started/installation). If the installer reports a missing prerequisite, install it and re-run the one-liner. The installer is idempotent.
{% endhint %}
### ⚡ Quickstart
After installing Hermes, we'll need to install Unsloth Studio to enable Hermes to serve and run inference of local models.
1. **Install or update Unsloth Studio.** Earlier versions don't expose the external API. See Installation.
2. **Launch Unsloth.** Note the port it starts on is usually `8000` or `8888`. You'll see it in the terminal output and in the browser URL (`http://localhost:PORT`).
3. **Load a model.** Click **New Chat**, pick or search a model (GGUF), and wait for it to finish loading.
4. **Connect Hermes.** Run `unsloth start hermes`. It mints an API key, writes the config, and launches Hermes against your loaded model.
#### ⚡ Connect with `unsloth start`
The fastest way to point Hermes at your local model is the `unsloth start` command. With Unsloth running and a model loaded, run:
```bash
unsloth start hermes
```
This mints an API key for you, writes the `unsloth` provider into `~/.hermes/config.yaml`, and launches Hermes against your loaded model. It replaces the `hermes setup` wizard below.
By default it uses the model already loaded in Unsloth. To load and use a specific model, pass `--model`:
```bash
unsloth start hermes --model unsloth/Qwen3.6-27B-GGUF
```
Connecting to Unsloth on another machine? Create a key (below) and pass it with `--api-key`, then point `UNSLOTH_STUDIO_URL` at the server.
The steps below set the same thing up manually with the `hermes setup` wizard, if you'd rather not use `unsloth start`.
### 🔑 Creating an API key
1. Open the sidebar, click your **Unsloth** avatar at the bottom-left.
2. Go to **Settings** → **API**.
3. Enter a friendly name (e.g. `hermes-agent-macbook`).
4. *(Optional)* Set an expiry.
5. Click **Create**.
6. **Copy the key immediately.** Unsloth stores only a hash and you won't be able to view it again.
All keys start with the `sk-unsloth-` prefix. Revoke a key from the same page at any time. Requests made with a revoked key will fail with `401 Unauthorized`.
### 🦥 Integrate Hermes with Unsloth API
Hermes sends each chat turn to a configured inference provider and connects to **OpenAI-compatible** endpoints. Configure the provider during installation or later in the setup wizard.
**1. Open the setup wizard:**
```bash
hermes setup
```
Pick **Model & Provider** from the "What would you like to do?" menu to configure only the inference endpoint, or **Full Setup** to walk through everything (TTS, tools, messaging gateway, agent settings).
**2. Select the Custom OpenAI-compatible endpoint** when Hermes prompts you for an inference provider.
**3. Fill in the prompts** as Hermes walks through them:
| Prompt | Value |
| ------------------------------------- | ---------------------------------------------------------- |
| **API base URL** | `http://localhost:8888/v1` *(your Unsloth port + `/v1`)* |
| **API key** | Your `sk-unsloth-…` key |
| **Detected model: … Use this model?** | `Y` *(Hermes auto-detects the model via `GET /v1/models`)* |
| **Context length in tokens** | *(leave blank for auto-detect)* |
| **Display name** | Anything you like, e.g. `unsloth-api` |
Hermes verifies the endpoint against `/v1/models` and confirms the detected model before continuing.
**4. Accept defaults for the remaining prompts** (TTS, tools, messaging gateway, agent settings) you can reconfigure any of them later. Hermes writes everything to `~/.hermes/config.yaml` and `~/.hermes/.env`.
**5. Launch Hermes:**
```bash
hermes
```
The startup banner shows your Unsloth model name in the status bar (e.g. `unsloth/Qwen3.6-27B-GGUF`), and the prompt is ready for input.
{% hint style="info" %}
To reconfigure just the model later, run `hermes setup model`. To edit the config file directly, `hermes config edit` opens `~/.hermes/config.yaml` in your `$EDITOR`.
{% endhint %}
### Optional: tune the Unsloth server
`unsloth run` starts the local API server and loads a model for your app to connect to. You can also customize how the server behaves when starting it.
```bash
# Serve Hermes (--disable-tools passes the agent's own tools through)
unsloth run \
--model unsloth/gemma-4-26B-A4B-it-GGUF \
--disable-tools \
--reasoning off \
-p 8888
```
{% hint style="warning" %}
Use `--disable-tools` when driving Hermes (or any external agent with its own tools). By default Unsloth Studio runs its own server-side tools, which swallows the agent's tool calls, so Hermes answers but never runs its tools. `--disable-tools` switches to passthrough, so Hermes's own tools are used.
{% endhint %}
Use `--reasoning off` to turn thinking off, or `--reasoning on` to turn it on for models that support reasoning.
```bash
# Expose the API on your local network
unsloth run \
--model unsloth/gemma-4-26B-A4B-it-GGUF \
-H 0.0.0.0 \
-p 8888
```
This starts the server on `0.0.0.0:8888`, allowing other devices on your local network to connect. `-p` changes which port the server runs on. If you want phones, laptops, or other devices on your network to connect to the API server, start it with `-H 0.0.0.0`.
Some apps may still override generation settings for individual requests. For more advanced runtime configuration, see the main [API tuning](https://unsloth.ai/docs/basics/api#unsloth-run-command) section.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://unsloth.ai/docs/integrations/hermes-agent.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.