### :cake:动态量化消融实验
我们还做了一些消融实验，以确认我们的校准数据集和动态量化方法是否真正有效。Unsloth 动态方法的诀窍在于将 **重要层量化为更高位宽** 例如 8bits，而 **不重要层则保留为较低位宽，比如 2bits**.
为测试我们的方法，我们让特定张量保持较低精度，例如 4bit 对比更高精度。下面的例子中，我们保留 `attn_k_b` 张量为 4bit（半动态）而非 8bit（Unsloth 当前方案）；而量化大小仅增加约 100MB（<0.1%）时，准确率就会大幅提升！
{% hint style="success" %}
`attn_k_b` 而 DeepSeek V3.1 中的其他张量对量化非常重要/敏感，应保留更高精度以维持准确率！
{% endhint %}
### :bug:聊天模板错误修复
在测试 DeepSeek-V3.1 量化方案时，我们发现一些低位宽量化方案没有正确包裹 `` ，或出现一些奇怪的格式。这导致某些社区量化方案在低位宽下无法工作，从而造成不公平比较。我们发现 llama.cpp 对 minja（一个更简单的 jinja 版本）的使用并不接受 `.split`中的位置参数。我们不得不将：
```
{%- set content = content.split("", 1)[1] -%}
```
改为如下：
```
{%- set splitted = content.split("") -%}
{%- set content = splitted[1:] | join("") -%}
```
请看 [这里](https://huggingface.co/unsloth/DeepSeek-V3.1-GGUF?chat_template=default\&format=true) 用于我们的固定聊天模板，或者 [这里](https://huggingface.co/unsloth/DeepSeek-V3.1/raw/main/chat_template.jinja) 用于原始 jinja 文件。
### :bar\_chart:通过率 1
Aider 主要报告通过率 2。我们也报告通过率 1，以比较相同大小的社区量化方案。我们看到，我们的动态量化在与相近大小的其他社区量化方案相比时表现更好，尤其是在小于 2 bit 和大于 4 bits 的情况下。3 bit 和 4 bit 的表现也同样出色。
## :computer:运行 DeepSeek V3.1 动态量化
前往我们的 [DeepSeek V3.1 指南](/docs/zh/mo-xing/tutorials/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit.md) 或者要快速获取动态 2bit 版本，请执行：
```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \\
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
```
然后使用 `llama.cpp` 直接下载权重。我们也已设置好推荐的最优参数，如 temperature、聊天模板等：
```bash
export LLAMA_CACHE="unsloth/DeepSeek-V3.1-GGUF"
./llama.cpp/llama-cli \\
-hf unsloth/DeepSeek-V3.1-GGUF:Q2_K_XL \\
--jinja \\
--n-gpu-layers 99 \
--temp 0.6 \\
--top-p 0.95 \\
--min-p 0.01 \
--ctx-size 8192 \
--seed 3407 \
-ot ".ffn_.*_exps.=CPU"
```
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://unsloth.ai/docs/zh/ji-chu-zhi-shi/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
