> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/zh/ji-chu-zhi-shi/continued-pretraining.md).

# 继续预训练

也称为持续微调。Unsloth 允许你持续进行预训练，使模型能够学习新语言。

* “ [文本补全笔记本](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_\(7B\)-Text_Completion.ipynb) 用于继续预训练/原始文本。
* “ [继续预训练笔记本](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_\(7B\)-CPT.ipynb) 用于学习另一种语言。

您可以在我们的 [博客文章](https://unsloth.ai/blog/contpretraining).

## 什么是继续预训练？

继续或持续预训练（CPT）对于“引导”语言模型理解新的知识领域或分布外领域是必要的。像 Llama-3 8b 或 Mistral 7b 这样的基础模型，首先会在包含万亿级 token 的巨大数据集上进行预训练（例如，Llama-3 有 15 万亿）。

但有时这些模型在其他语言或特定文本领域上训练得并不好，比如法律、医学或其他领域。因此，需要进行继续预训练（CPT），让语言模型学习新的 token 或数据集。

## 高级功能：

### 为继续微调加载 LoRA 适配器

如果您通过 Unsloth 保存了一个 LoRA 适配器，您也可以使用您的 LoRA 权重继续训练。优化器状态也会被重置。若要连同优化器状态一起加载以继续微调，请参见下一节。

```python
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "LORA_MODEL_NAME",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)
trainer = Trainer(...)
trainer.train()
```

### 继续预训练与微调 `lm_head` 和 `embed_tokens` 矩阵

添加 `lm_head` 和 `embed_tokens`。对于 Colab，有时 Llama-3 8b 会内存不足。如果是这样，只需添加 `lm_head`.

```python
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",
                      "lm_head", "embed_tokens",],
    lora_alpha = 16,
)
```

然后使用 2 个不同的学习率——其中一个比 `lm_head` 或 `embed_tokens` 如下所示：

```python
from unsloth import UnslothTrainer, UnslothTrainingArguments

trainer = UnslothTrainer(
    ....
    args = UnslothTrainingArguments(
        ....
        learning_rate = 5e-5,
        embedding_learning_rate = 5e-6, # 比 learning_rate 小 2-10 倍
    ),
)
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://unsloth.ai/docs/zh/ji-chu-zhi-shi/continued-pretraining.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.