| 全景放射图显示右侧后上颌骨有一处骨溶解性病灶，并伴有上颌窦底壁吸收（箭头所示）。 |
要对数据集进行格式化，所有视觉微调任务都应按以下方式格式化：
```python
[
{ "role": "user",
"content": [{"type": "text", "text": instruction}, {"type": "image", "image": image} ]
},
{ "role": "assistant",
"content": [{"type": "text", "text": answer} ]
},
]
```
我们将编写一个自定义指令，要求 VLM 扮演放射科专家。也请注意，除了只使用 1 条指令之外，你还可以添加多轮对话，使其成为动态对话。
{% code expandable="true" %}
```notebook-python
instruction = "你是一名放射科专家。请准确描述你在这张图像中看到的内容。"
def convert_to_conversation(sample):
conversation = [
{ "role": "user",
"content" : [
{"type" : "text", "text" : instruction},
{"type" : "image", "image" : sample["image"]} ]
},
{ "role" : "assistant",
"content" : [
{"type" : "text", "text" : sample["caption"]} ]
},
]
return { "messages" : conversation }
pass
```
{% endcode %}
让我们把数据集转换为用于微调的“正确”格式：
```notebook-python
converted_dataset = [convert_to_conversation(sample) for sample in dataset]
```
第一个示例如下所示：
```notebook-python
converted_dataset[0]
```
{% code overflow="wrap" %}
```
{'messages': [{'role': 'user',
'content': [{'type': 'text',
'text': '你是一名放射科专家。请准确描述你在这张图像中看到的内容。'},
{'type': 'image',
'image': }]},
{'role': 'assistant',
'content': [{'type': 'text',
'text': '全景放射图显示右侧后上颌骨有一处骨溶解性病灶，并伴有上颌窦底壁吸收（箭头所示）。'}]}]}
```
{% endcode %}
在进行任何微调之前，也许视觉模型已经知道如何分析图像了？让我们来检查一下是否如此！
{% code expandable="true" %}
```notebook-python
FastVisionModel.for_inference(model) # 启用推理！
image = dataset[0]["image"]
instruction = "你是一名放射科专家。请准确描述你在这张图像中看到的内容。"
messages = [
{"role": "user", "content": [
{"type": "image"},
{"type": "text", "text": instruction}
]}
]
input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True)
inputs = tokenizer(
image,
input_text,
add_special_tokens = False,
return_tensors = "pt",
).to("cuda")
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer, skip_prompt = True)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,
use_cache = True, temperature = 1.5, min_p = 0.1)
```
{% endcode %}
结果如下：
```
这张放射影像似乎是上、下牙列的全景图，具体来说是全景口腔片（OPG）。
* 全景放射图显示正常的牙科结构。
* 右上方存在一个异常区域，表现为骨质透亮区，对应于上颌窦。
**关键观察**
* 左上牙之间的骨质相对更为致密。
* 图像上方有两个大箭头，表明需要更仔细地检查该区域。其中一个箭头位于左侧位置，另一个位于右侧位置。然而，只有
```
更多详情，请查看我们的数据集部分中的 [笔记本这里](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(11B\)-Vision.ipynb#scrollTo=vITh0KVJ10qX).
### :mag\_right:仅对视觉模型、VLM 训练助手回复
对于语言模型，我们可以使用 `from unsloth.chat_templates import train_on_responses_only` 如前所述。对于视觉模型，请将额外参数作为 `UnslothVisionDataCollator` 和之前一样！请参见 [#vision-data-collator](#vision-data-collator "mention") 以了解有关如何使用视觉数据整理器的更多细节。
{% code overflow="wrap" %}
```python
class UnslothVisionDataCollator:
def __init__(
self,
...
# from unsloth.chat_templates import train_on_responses_only
# trainer = train_on_responses_only(
# trainer,
# instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n",
# response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n",
# )
train_on_responses_only = False, # 对于 LLM 来说等同于 train_on_responses_only
instruction_part = None, # 等同于 train_on_responses_only(instruction_part = ...)
response_part = None, # 等同于 train_on_responses_only(response_part = ...)
force_match = True, # 也匹配换行符！
)
```
{% endcode %}
例如，针对 Llama 3.2 Vision：
```python
UnslothVisionDataCollator(
model, tokenizer,
...
train_on_responses_only = True,
instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n",
response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n",
...
)
```
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://unsloth.ai/docs/zh/ji-chu-zhi-shi/vision-fine-tuning.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
