> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/zh/kuai-su-kai-shi/reinforcement-learning-rl-guide/advanced-rl-documentation/gspo-reinforcement-learning.md). # GSPO 强化学习在 Unsloth 中使用 GSPO（Group Sequence Policy Optimization，组序列策略优化）进行 RL 训练。我们正在介绍 GSPO，它是……的一种变体 [GRPO](/docs/zh/kuai-su-kai-shi/reinforcement-learning-rl-guide.md#from-rlhf-ppo-to-grpo-and-rlvr) 由阿里巴巴的 Qwen 团队提出。他们注意到这样一个现象：当 GRPO 对每个 token 采用重要性权重时，尽管优势本质上并不会随着每个 token 缩放或变化。由此促成了 GSPO 的诞生；它现在把重要性赋予序列似然，而不是各个 token 的单独似然。 * 使用我们免费的 GSPO 笔记本来： [**gpt-oss-20b**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) 和 [**Qwen2.5-VL**](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_5_7B_VL_GRPO.ipynb) 通过设置以下内容在 Unsloth 中启用 GSPO `importance_sampling_level = "sequence"` 在 GRPO 配置中。这两种算法之间的区别如下所示，均来自 Qwen 和阿里巴巴的 GSPO 论文：

在公式 1 中可以看出，优势会在该张量求和之前将每一行缩放到 token 的 logprob 中。本质上，尽管该缩放是针对整个序列而不是每个单独 token 给出的，但每个 token 得到的缩放都是相同的。下方可见一个简单示意图：

公式 2 表明，在计算出 Logprob 比率后，每个序列的 logprob 比率会先求和并指数化，而只有得到的序列比率会再按行乘以优势。

启用 GSPO 很简单，你只需要设置 `importance_sampling_level = "sequence"` 在 GRPO 配置中的该标志。 ```python training_args = GRPOConfig( output_dir = "vlm-grpo-unsloth", per_device_train_batch_size = 8, gradient_accumulation_steps = 4, learning_rate = 5e-6, adam_beta1 = 0.9, adam_beta2 = 0.99, weight_decay = 0.1, warmup_ratio = 0.1, lr_scheduler_type = "cosine", optim = "adamw_8bit", # beta = 0.00, epsilon = 3e-4, epsilon_high = 4e-4, num_generations = 8, max_prompt_length = 1024, max_completion_length = 1024, log_completions = False, max_grad_norm = 0.1, temperature = 0.9, # report_to = "none", # 如果你想记录到 Weights & Biases，则设为 "wandb" num_train_epochs = 2, # 用于快速测试运行，完整训练时增大 # GSPO 如下： importance_sampling_level = "sequence", # Dr GRPO / GAPO 等 loss_type = "dr_grpo", ) ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://unsloth.ai/docs/zh/kuai-su-kai-shi/reinforcement-learning-rl-guide/advanced-rl-documentation/gspo-reinforcement-learning.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.