> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/models/nemotron-3.5-bolt.md).
# NVIDIA Nemotron 3.5 Bolt: Run Locally Guide
Run Nemotron 3.5 Bolt locally with Unsloth GGUFs.
NVIDIA Nemotron-3.5-Bolt-74B-A9B is a **74B parameter, 9B active** hybrid reasoning MoE model compressed from Super VL. It supports **text, images and video**, tool calling and a **128K context** window. Unsloth's **1-bit and 2-bit GGUFs** range from **20.7GB to 23.6GB** and can run locally in Unsloth Desktop or llama.cpp.
Its **Hybrid LatentMoE** architecture combines Mamba-2, attention and Mixture-of-Experts layers. NVIDIA's IQ2 release stores most expert weights at **2.06-bit IQ2\_XXS**, while keeping Mamba layers in NVFP4 and the output head in Q8\_0.
We further quantized the model as well - our UD-IQ2\_XXS is the exact duplicate of the official IQ2\_XXS except we quantize the embedding layer from F32 to Q8\_0, saving some space. **Unsloth quants:** [NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF)
### **⚙️ Usage Guide**
**Nemotron 3.5 Bolt Requirements**
For short-context use, budget around **32GB RAM or unified memory**; **48–64GB** gives more headroom. Bolt is available in **1-bit and 2-bit**:
| Quantization | Download size | Est. minimum RAM |
| ------------------- | ------------- | ---------------- |
| 1-bit · UD-IQ1\_S | 20.7GB | 24GB |
| 1-bit · UD-IQ1\_M | 22.5GB | 32GB |
| 2-bit · UD-IQ2\_XXS | 23.1GB | 32GB |
| 2-bit · IQ2\_XXS | 23.6GB | 32GB |
* Memory figures are estimates for short-context use. Longer contexts and images/video need extra memory for the cache and vision projector. Download sizes are for the GGUF weights.
{% hint style="info" %}
Nemotron 3.5 Bolt uses a **DSpark** draft model in **NVFP4** (833MB) by default in Unsloth for faster output. It needs about 1–2GB more memory.
{% endhint %}
#### Recommended Settings
Unsloth automatically configures inference settings. The llama.cpp examples use default sampling settings because NVIDIA has not specified Bolt-specific temperature or top-p values.
* **Context:** Start with 4K and increase as memory allows, up to 128K tokens.
* **Chat template:** Use the template included with the model.
### Run Nemotron 3.5 Bolt Guide
We'll use the **23.1GB `UD-IQ2_XXS`** quant in this tutorial. Feel free to choose another quant from the table above.
#### 🦥 Run Nemotron 3.5 Bolt in Unsloth Desktop
Run Nemotron 3.5 Bolt directly in [Unsloth Desktop](https://unsloth.ai/docs/desktop) on **macOS, Windows and Linux**. Search, download and chat with the model in the app.
{% stepper %}
{% step %}
#### **Install Unsloth**
The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop.md).
Download Unsloth
* :apple: [Download for macOS](https://unsloth.ai/download/mac)
* :windows: [Download for Windows](https://unsloth.ai/download/windows)
* :linux: [Download for Linux](https://unsloth.ai/download/linux)
You can also install manually:
{% tabs %}
{% tab title="Native install" %}
**MacOS, Linux, WSL:**
```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```
**Windows:**
```powershell
irm https://unsloth.ai/install.ps1 | iex
```
{% endtab %}
{% tab title="Docker" %}
The [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth) `unsloth/unsloth` is available on Docker. [Read guide.](https://unsloth.ai/docs/get-started/install/docker)
{% endtab %}
{% tab title="Homebrew" %}
Install Unsloth Desktop:
{% code collapsedlinecount="10" %}
```bash
brew install --cask unsloth
```
{% endcode %}
Update Unsloth Desktop:
```bash
brew upgrade --cask unsloth
```
{% endtab %}
{% tab title="WinGet" %}
Install Unsloth Desktop:
```powershell
winget install --id UnslothAI.Unsloth -e
```
Update Unsloth Desktop:
```powershell
winget upgrade --id UnslothAI.Unsloth -e
```
{% endtab %}
{% endtabs %}
{% endstep %}
{% step %}
#### **Setup and Run Nemotron 3.5 Bolt**
Go to [Unsloth Chat](/docs/new/studio/chat.md) or Model hub and search for **Nemotron 3.5 Bolt** in the search bar and download your desired model and quant. Now you're ready to run the model!
Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.
To try image understanding, attach a PNG or JPEG and ask **“What is this image?”**
For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat.md).
{% endstep %}
{% step %}
Serve Nemotron 3.5 Bolt with Unsloth API
Serve Nemotron 3.5 Bolt with `unsloth run`. You can pass llama-server flags for context size, GPU layers and other settings. See our [API docs](https://unsloth.ai/docs/basics/api) or [unsloth start](https://unsloth.ai/docs/integrations/unsloth-start).
{% code overflow="wrap" %}
```bash
unsloth run --model unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF:UD-IQ2_XXS \
--ctx-size 4096 --parallel 1 --alias nemotron-bolt \
--host 127.0.0.1 --port 8001
```
{% endcode %}
{% endstep %}
{% step %}
#### Unsloth is now ready
You can also do many other things with Nemotron 3.5 Bolt via Unsloth Desktop like:
* **Connect tools:** [Claude Code](/docs/basics/claude-code.md), [Codex](/docs/basics/codex.md), [web search](/docs/new/studio/chat.md#advanced-web-search), [MCP](/docs/basics/mcp.md) and more
* **Train models:** Fine-tune text, diffusion, [embedding](/docs/basics/embedding-finetuning.md), and more
* **Generate media:** Create and train [images](/docs/basics/diffusion-image.md), video, [TTS](/docs/basics/text-to-speech-tts-fine-tuning.md) locally
{% endstep %}
{% endstepper %}
### KLD Benchmarks
Bolt is already quantized at IQ2\_XXS, so we only made some smaller quants. UD-IQ2\_XXS is a direct replica of the official IQ2\_XXS quant, except we make the embedding layer Q8\_0 instead of F32. The smaller quants like IQ1\_S further quantize some of MoE experts down, saving around 2-3GB. This fits well for 24GB cards.
#### 🦙 Run Nemotron 3.5 Bolt in llama.cpp
{% stepper %}
{% step %}
**Build llama.cpp**
Install the latest version of llama.cpp. You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.
```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```
Video needs both `ffmpeg` and `ffprobe` on your PATH. Continue the commands below from this `llama.cpp` directory.
{% endstep %}
{% step %}
**Download the Unsloth GGUF**
Download `UD-IQ2_XXS` and the vision projector for image/video input. Change the quant name below to download another version:
{% code overflow="wrap" %}
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -U huggingface_hub
hf download unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF \
--local-dir models/nemotron-bolt \
--include "*UD-IQ2_XXS*.gguf" "*mmproj*.gguf"
```
{% endcode %}
Set the paths to the downloaded model and its matching projector. For a split GGUF, keep every shard together and point to the **first shard**. Adjust the filenames below to match your download.
```bash
MODEL_FILE="models/nemotron-bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-UD-IQ2_XXS.gguf"
MMPROJ_FILE="models/nemotron-bolt/mmproj-BF16.gguf"
```
{% endstep %}
{% step %}
**Run text and images**
For text:
```bash
./llama.cpp/llama-cli \
--model "$MODEL_FILE" --ctx-size 4096 \
--prompt "Explain what a vision-language model does."
```
For an image, set the path to your own PNG or JPEG:
{% code overflow="wrap" %}
```bash
./llama.cpp/llama-mtmd-cli \
--model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
--ctx-size 4096 \
--image /path/to/picture.jpg \
--prompt "Describe this image and extract any visible text."
```
{% endcode %}
{% endstep %}
{% step %}
**Run video**
Start the multimodal CLI:
```bash
./llama.cpp/llama-mtmd-cli \
--model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
--ctx-size 4096
```
Then enter these lines **inside the CLI**:
```
/video /path/to/video.mp4
Summarize the video and describe the key events.
```
Supports MP4 clips up to 2 minutes. Use up to 1 FPS / 128 frames at 1080p, or 2 FPS / 256 frames at 720p. Start with a short clip and increase context as needed.
{% endstep %}
{% endstepper %}
### Benchmarks
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://unsloth.ai/docs/models/nemotron-3.5-bolt.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
