> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/vllm-guide.md).

# vLLM 部署与推理指南

关于将 LLM 保存并部署到 vLLM 以在生产环境中提供 LLM 服务的指南

### :computer:安装 vLLM

对于 NVIDIA GPU，请使用 uv 并运行：

```bash
pip install --upgrade pip
pip install uv
uv pip install -U vllm --torch-backend=auto
```

对于 AMD GPU，请使用 nightly Docker 镜像： `rocm/vllm-dev:nightly`

对于 NVIDIA GPU 的 nightly 分支，请运行：

{% code overflow="wrap" %}

```bash
pip install --upgrade pip
pip install uv
uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
```

{% endcode %}

请看 [vLLM 文档](https://docs.vllm.ai/en/stable/getting_started/installation) 了解更多细节

### :truck:部署 vLLM 模型

保存好微调结果后，你只需执行：

```bash
vllm serve unsloth/gpt-oss-120b
```

### :fire\_engine:vLLM 部署服务器标志、引擎参数和选项

一些重要的服务器标志位可在 [#vllm-deployment-server-flags-engine-arguments-and-options](#vllm-deployment-server-flags-engine-arguments-and-options "mention")

### 🦥在 vLLM 中部署 Unsloth 微调模型

微调后 [Fine-tuning Guide](/docs/zh/kuai-su-kai-shi/fine-tuning-llms-guide.md) 或使用我们位于 [Unsloth 笔记本](/docs/zh/kuai-su-kai-shi/unsloth-notebooks.md)，你可以在单个工作流中直接通过 vLLM 保存或部署你的模型。比如，一个 Unsloth 微调脚本示例如下：

```python
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/gpt-oss-20b",
    max_seq_length = 2048,
    load_in_4bit = True,
)
model = FastLanguageModel.get_peft_model(model)
```

**要保存为 16 位以供 vLLM 使用，请使用：**

{% code overflow="wrap" %}

```python
model.save_pretrained_merged("finetuned_model", tokenizer, save_method = "merged_16bit")
## 或上传到 HuggingFace：
model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_16bit", token = "")
```

{% endcode %}

**若只保存 LoRA 适配器**，可使用：

```python
model.save_pretrained("finetuned_lora")
tokenizer.save_pretrained("finetuned_lora")
```

或者直接使用我们内置的函数来完成：

{% code overflow="wrap" %}

```python
model.save_pretrained_merged("finetuned_model", tokenizer, save_method = "lora")
## 或上传到 HuggingFace
model.push_to_hub_merged("hf/model", tokenizer, save_method = "lora", token = "")
```

{% endcode %}

要合并为 4bit 以便在 HuggingFace 上加载，首先调用 `merged_4bit`。然后使用 `merged_4bit_forced` 如果你确定要合并为 4bit。我非常不建议这样做，除非你清楚要如何使用这个 4bit 模型（例如用于 DPO 训练，或用于 HuggingFace 的在线推理引擎）

{% code overflow="wrap" %}

```python
model.save_pretrained_merged("finetuned_model", tokenizer, save_method = "merged_4bit")
## 上传到 HuggingFace：
model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_4bit", token = "")
```

{% endcode %}

然后在另一个终端中将微调后的模型加载到 vLLM：

```bash
vllm serve finetuned_model
```

如果上面的方法不起作用，你可能需要提供完整路径，例如：

```bash
vllm serve /mnt/disks/daniel/finetuned_model
```

查看其他内容：

### [vLLM 引擎参数](/docs/zh/ji-chu-zhi-shi/inference-and-deployment/vllm-guide/vllm-engine-arguments.md)

### [LoRA 热插拔指南](/docs/zh/ji-chu-zhi-shi/inference-and-deployment/vllm-guide/lora-hot-swapping-guide.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/zh/ji-chu-zhi-shi/inference-and-deployment/vllm-guide.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.