> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/multi-gpu-training-with-unsloth.md).

# 使用 Unsloth 进行多 GPU 微调

了解如何使用 Unsloth 在多块 GPU 上对 LLM 进行微调以及并行化。

Unsloth 目前通过 Accelerate 和 DeepSpeed 等库支持多 GPU 设置。这意味着你已经可以利用如下并行方法 **FSDP** 和 **DDP** 使用 Unsloth。

#### **请查看我们新的分布式数据并行** [**（DDP）多 GPU 指南**](/docs/zh/ji-chu-zhi-shi/multi-gpu-training-with-unsloth/ddp.md)**.**

我们知道这个过程可能很复杂，而且需要手动设置。我们正在努力让多 GPU 支持变得更加简单、更加用户友好，并且我们很快就会宣布 Unsloth 的官方多 GPU 支持。

目前，你可以使用我们的 [Magistral-2509 Kaggle 笔记本](/docs/zh/mo-xing/tutorials/magistral-how-to-run-and-fine-tune.md#fine-tuning-magistral-with-unsloth) 作为一个示例，它使用多 GPU Unsloth 来适配 24B 参数模型，或者我们的 [DDP 指南](/docs/zh/ji-chu-zhi-shi/multi-gpu-training-with-unsloth/ddp.md).

**与此同时**，要为 DDP 启用多 GPU，请执行以下操作：

1. 将你的训练脚本创建为 `train.py` （或类似名称）。例如，你可以使用我们的其中一个 [训练脚本](https://github.com/unslothai/notebooks/tree/main/python_scripts) 这些脚本是从我们的各种笔记本创建的！
2. 运行 `accelerate launch train.py` 或 `torchrun --nproc_per_node N_GPUS train.py` 其中 `N_GPUS` 是你拥有的 GPU 数量。

#### **流水线 / 模型拆分加载**

如果单个 GPU 没有足够的 VRAM 来加载例如 Llama 70B，也不用担心——我们会为你把模型分到每个 GPU 上！要启用此功能，请使用 `device_map = "balanced"` 标志：

```python
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.3-70B-Instruct",
    load_in_4bit = True,
    device_map = "balanced",
)
```

**敬请期待我们的正式公告！**\
更多详情，请查看我们正在进行的 [拉取请求](https://github.com/unslothai/unsloth/issues/2435) 关于多 GPU 支持的讨论。


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/zh/ji-chu-zhi-shi/multi-gpu-training-with-unsloth.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.