> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/zh/kuai-su-kai-shi/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking.md).
# RL 奖励黑客
了解强化学习中的奖励黑客是什么，以及如何应对它。
强化学习（RL）的终极目标是最大化某种奖励（比如速度、收入或某个指标）。但 RL 可能会 **作弊。** 当 RL 算法学会了某种技巧或利用某些漏洞来提高奖励，但实际上并没有完成最终任务时，这被称为“**奖励黑客**".
这就是模型学会修改单元测试以通过编程挑战的原因，而这也是实际部署中的关键阻碍。还有一些很好的例子来自 [维基百科](https://en.wikipedia.org/wiki/Reward_hacking).
**能对抗奖励黑客吗？可以！** 在我们的 [免费的 gpt-oss 强化学习笔记本](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-\(20B\)-GRPO.ipynb) 中，我们探讨了如何在代码生成场景中对抗奖励黑客，并展示了针对常见错误模式的切实解决方案。我们看到模型会修改计时函数、把工作外包给其他库、缓存结果，甚至直接作弊。在进行对抗后，结果是我们的模型生成的是真正优化过的矩阵乘法内核，而不是花招式的作弊。
## :trophy: 奖励黑客概述
RL 期间奖励黑客的一些常见例子包括：
#### 偷懒
RL 学会使用 Numpy、Torch 及其他库，而这些库会调用经过优化的 CUDA 内核。我们可以通过检查生成的代码是否导入了其他非标准 Python 库，来阻止 RL 算法调用优化代码。
#### 缓存与作弊
RL 学会缓存输出结果，并学会通过检查 Python 全局变量来找到实际输出。
我们可以用一个巨大的假矩阵清空缓存，从而阻止 RL 算法使用缓存数据。我们还必须通过多次循环和多轮测试来谨慎地进行基准测试。
#### 作弊
RL 学会修改计时函数，使其输出 0 时间作为经过时间。我们可以通过限制它的 `局部变量` 和 `全局变量`。我们还将使用 `exec` 来创建该函数，因此我们必须将输出保存到一个空字典中。我们还通过以下方式禁止访问全局变量： `types.FunctionType(f.__code__, {})`\\\\
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://unsloth.ai/docs/zh/kuai-su-kai-shi/reinforcement-learning-rl-guide/advanced-rl-documentation/rl-reward-hacking.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
