{% endstep %}
{% endstepper %}
### 🦙 Llama.cpp 指南
对于本指南，无需选择量化类型，因为只有一种： `UD-Q4_K_XL`。参见： [Gemma 4 QAT 集合](https://huggingface.co/collections/unsloth/gemma-4-qat)。对于这些教程，我们将使用 [llama.cpp](llama.cpphttps://github.com/ggml-org/llama.cpp) 进行快速本地推理，尤其是如果你有 CPU。
{% stepper %}
{% step %}
获取最新的 `llama.cpp` **在** [**GitHub 上的这里**](https://github.com/ggml-org/llama.cpp)。你也可以按照下面的构建说明进行。将 `-DGGML_CUDA=ON` 改为 `-DGGML_CUDA=OFF` ，如果你没有 GPU，或者只是想进行 CPU 推理。 **对于 Apple Mac / Metal 设备**，设置 `-DGGML_CUDA=OFF` ，然后像平常一样继续——Metal 支持默认开启。
```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \\
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```
{% endstep %}
{% step %}
如果你想 `llama.cpp` 直接加载模型，可以根据每个模型按照下面的命令操作。 `UD-Q4_K_XL` 是唯一的量化类型。你也可以通过 Hugging Face 下载（第 3 步）。这类似于 `ollama run` 。使用 `export LLAMA_CACHE="folder"` 来强制 `llama.cpp` 保存到特定位置。
**26B-A4B：**
```bash
export LLAMA_CACHE="unsloth/gemma-4-26B-A4B-it-qat-GGUF"
./llama.cpp/llama-cli \\
-hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL \\
--temp 1.0 \\
--top-p 0.95 \\
--top-k 64
```
**31B：**
```bash
export LLAMA_CACHE="unsloth/gemma-4-31B-it-qat-GGUF"
./llama.cpp/llama-cli \\
-hf unsloth/gemma-4-31B-it-qat-GGUF:UD-Q4_K_XL \\
--temp 1.0 \\
--top-p 0.95 \\
--top-k 64
```
**E4B：**
```bash
export LLAMA_CACHE="unsloth/gemma-4-E4B-it-qat-GGUF"
./llama.cpp/llama-cli \\
-hf unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL \\
--temp 1.0 \\
--top-p 0.95 \\
--top-k 64
```
**E2B：**
```bash
export LLAMA_CACHE="unsloth/gemma-4-E2B-it-GGUF"
./llama.cpp/llama-cli \\
-hf unsloth/gemma-4-E2B-it-qat-GGUF:UD-Q4_K_XL \\
--temp 1.0 \\
--top-p 0.95 \\
--top-k 64
```
{% endstep %}
{% step %}
通过以下方式下载模型（在安装 `pip install huggingface_hub hf_transfer` 之后）。你可以选择 `UD-Q4_K_XL` 或其他量化版本，例如 `Q8_0` 。如果下载卡住，请参见： [Hugging Face Hub、XET 调试](/docs/zh/ji-chu/troubleshooting-and-faqs/hugging-face-hub-xet-debugging.md)
```bash
hf download unsloth/gemma-4-26B-A4B-it-qat-GGUF \\
--local-dir unsloth/gemma-4-26B-A4B-it-qat-GGUF \\
--include "*mmproj-BF16*" \\
--include "*UD-Q4_K_XL*" # 动态 2bit 请使用 "*UD-Q2_K_XL*"
```
{% endstep %}
{% step %}
然后以对话模式运行模型（带视觉 `mmproj-F16`):
{% code overflow="wrap" %}
```bash
./llama.cpp/llama-cli \\
--model unsloth/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \\
--mmproj unsloth/gemma-4-26B-A4B-it-qat-GGUF/mmproj-BF16.gguf \\
--temp 1.0 \\
--top-p 0.95 \\
--top-k 64
```
{% endcode %}
{% endstep %}
{% step %}
### Llama-server 部署
要在 llama-server 上部署 Gemma-4，请使用：
```bash
./llama.cpp/llama-server \\
--model unsloth/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \\
--mmproj unsloth/gemma-4-26B-A4B-it-qat-GGUF/mmproj-BF16.gguf \\
--temp 1.0 \\
--top-p 0.95 \\
--top-k 64 \\
--alias "unsloth/gemma-4-26B-A4B-it-qat-GGUF" \\
--port 8001 \\
--chat-template-kwargs '{"enable_thinking":true}'
```
{% hint style="warning" %}
要 [禁用思考 / 推理](#how-to-enable-or-disable-reasoning-and-thinking)，使用 `--chat-template-kwargs '{"enable_thinking":false}'`
如果你使用的是 **Windows** PowerShell，请使用： `--chat-template-kwargs "{\"enable_thinking\":false}"`
可将 'true' 和 'false' 互换使用。
{% endhint %}
{% endstep %}
{% endstepper %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://unsloth.ai/docs/zh/mo-xing/gemma-4/qat.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.