> For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unsloth.ai/docs/models/deepseek-v4.md). # DeepSeek-V4: How to Run Locally DeepSeek-V4 is DeepSeek's new open models including **DeepSeek-V4-Pro** with **1.6T** parameters (49B active) and **DeepSeek-V4-Flash** with **284B** parameters (13B active). The models excel at coding, agentic workflows and chat with a **1M context** window. In this guide we'll show you how to run DeepSeek-V4-Flash locally. For **lossless** DeepSeek, use Q8 (`UD-Q8_K_XL`), which is is **162 GB and** only **7GB larger** than Q4 (`UD-Q4_K_XL`). DeepSeek-V4-Flash scores 86.2% on MMLU-Pro, 88.1% on GPQA Diamond, and 56.9% on Terminal Bench 2.0. [DeepSeek-V4-Flash GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF) {% hint style="warning" %} **WIP:** Smaller quants are still a work in progress, will be released tomorrow. {% endhint %} Usage GuideRunning Tutorials ### 📊 Quantization Analysis DeepSeek-V4-Flash stores its routed-expert weights natively in **FP4** (e2m1 values with e8m0 block scales, the MX "MXFP4" format). Everything else (attention, shared experts, dense layers) is stored in FP8 (e4m3 with 128x128 ue8m0 block scales) or BF16. **Non**-Unsloth DeepSeek-V4-Flash GGUFs were converted without these paths thus deviating from the official weights. Unsloth's `UD-Q8_K_XL` keeps the routed experts in native MXFP4 (bit-exact repack of the official FP4 weights) and stores every FP8-origin tensor as its exact BF16 dequantization. We verified this against the official `deepseek-ai/DeepSeek-V4-Flash` weights, tensor by tensor: **all 1,328 tensors are bit-identical.** The weights match (RMSE = 0.0). **UD-Q8\_K\_XL is the lossless model, in GGUF form.** The histogram below shows the FP4 code usage of a routed-expert tensor (blk.30.ffn\_down\_exps, expert 0). The model lives on the e2m1 grid, using all 16 codes, including both +0 and -0.
#### ⚙️ Usage Guide DeepSeek-V4-Flash is the smaller and faster than DeepSeek-V4-Pro, with **284B** parameters (13B active), and a **1M context window**. The model has 3 modes, **Non-think**, **Think** **High** and **Think** **Max**. The `UD-Q8_K_XL` quant is DeepSeek-V4-Flash in full original precision. It is 162GB size and it's best to have at least 168GB of available RAM/VRAM available. **Table: Inference hardware requirements** (units = total memory: RAM + VRAM, or unified memory) | 2-bit | 3-bit | 4-bit (near Lossless) | Q8\_K\_XL (Lossless) | | ----- | ----- | --------------------- | -------------------- | | TBD | TBD | 162 GB | 168 GB | {% hint style="success" %} For best performance, make sure your total available memory, including VRAM and system RAM, exceeds the quantized model file size by a comfortable margin. {% endhint %} #### Recommended Settings DeepSeek recommends these parameters for best performance: `temperature=1.0`, `top-p=1.0` **Think High is on by default.** If disabled, you can enable it via: `--chat-template-kwargs '{"enable_thinking":true}'` or toggle it via the UI dropdown in [Unsloth Studio](#unsloth-studio-guide). {% columns %} {% column %} | `temperature = 1.0` | | ------------------- | | `top-p = 1.0` | | {% endcolumn %} | {% column %} * **Maximum context window:** `1,048,576` * For Think Max, set context to at least **384K tokens**. {% endcolumn %} {% endcolumns %} ## Run DeepSeek-V4-Flash Tutorials: For this tutorial, we will use the 8-bit quant `UD-Q8_K_XL`, as it fits on a 170GB RAM device. You can now run DeepSeek-V4-Flash in [Unsloth Studio](#run-in-unsloth-studio). 🦥 Unsloth Studio Guide🦙 Llama.cpp Guide ### 🦥 Unsloth Studio Guide DeepSeek-V4-Flash can now be run and trained in [Unsloth Studio](/docs/new/studio.md), our new open-source web UI for local AI. Unsloth Studio lets you run models locally on **MacOS**, **Windows**, Linux and: {% columns %} {% column %} * Search, download, [run GGUFs](/docs/new/studio.md#run-models-locally) and safetensor models * [**Self-healing** tool calling](/docs/new/studio.md#execute-code--heal-tool-calling) + **web search** * [**Code execution**](/docs/new/studio.md#run-models-locally) (Python, Bash) * [Automatic inference](/docs/new/studio.md#model-arena) parameter tuning (temp, top-p, etc.) * Fast CPU + GPU inference via llama.cpp * [Train LLMs](/docs/new/studio.md#no-code-training) 2x faster with 70% less VRAM {% endcolumn %} {% column %}
{% endcolumn %} {% endcolumns %} {% stepper %} {% step %} #### Install Unsloth Ensure you use the latest version. Run in your terminal: **MacOS, Linux, WSL:** ```bash curl -fsSL https://unsloth.ai/install.sh | sh ``` **Windows PowerShell:** ```bash irm https://unsloth.ai/install.ps1 | iex ``` {% endstep %} {% step %} #### Launch Unsloth **MacOS, Linux, WSL and Windows:** ```bash unsloth studio -H 0.0.0.0 -p 8888 ``` Then open `http://127.0.0.1:8888` (or your specific URL) in your browser. {% endstep %} {% step %} #### Search and download DeepSeek-V4-Flash On first launch you will need to create a password to secure your account and sign in again. Then go to the [Studio Chat](/docs/new/studio/chat.md) tab and search for DeepSeek-V4-Flash in the search bar and download your desired model and quant.
{% endstep %} {% step %} #### Run DeepSeek-V4-Flash Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. Because **Think High is on by default**, you can go to the right dropdown to toggle it to Non-think or Think Max. You can also edit the context length, chat template and other settings. For more information, you can view our [Unsloth Studio inference guide](/docs/new/studio/chat.md).
{% endstep %} {% endstepper %} ### 🦙 Llama.cpp Guide {% stepper %} {% step %} Obtain the latest `llama.cpp` **on** [**GitHub here**](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default. ```bash apt-get update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggml-org/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split cp llama.cpp/build/bin/llama-* llama.cpp ``` {% endstep %} {% step %} You can now use `llama.cpp` directly to load and download models, just like `ollama run`. First, select the quantization type you want like `Q8_K_XL`. Also use `export LLAMA_CACHE="folder"` to force `llama.cpp` to save to a specific location. Note this download process might be very slow, so it's probably best to use the manual download process in the next section. ```bash export LLAMA_CACHE="unsloth/DeepSeek-V4-Flash-GGUF" ./build/bin/llama-cli \ -hf unsloth/DeepSeek-V4-Flash-GGUF:UD-Q8_K_XL \ --temp 1.0 \ --top-p 1.0 ``` {% endstep %} {% step %} If you want to download the model manually, we can download the model via the code below (after installing `pip install huggingface_hub`). If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging.md) ```bash hf download unsloth/DeepSeek-V4-Flash-GGUF \ --local-dir unsloth/DeepSeek-V4-Flash-GGUF \ --include "*UD-Q8_K_XL*" # Use "*UD-IQ4_XS*" for 4-bit ``` {% endstep %} {% step %} You can edit `--threads 32` for the number of CPU threads, `--ctx-size 32768` for context length, `--n-gpu-layers 2` for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. {% code overflow="wrap" %} ```bash ./build/bin/llama-cli \ --model unsloth/DeepSeek-V4-Flash-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-UD-Q8_K_XL-00001-of-00004.gguf \ --temp 1.0 \ --top-p 1.0 ``` {% endcode %} {% endstep %} {% endstepper %} ## 📊 Benchmarks See further below for benchmarks in table format:
| Benchmark (Metric) | V4-Flash Non-Think | V4-Flash High | V4-Flash Max | V4-Pro Non-Think | V4-Pro High | V4-Pro Max | | --------------------------- | :----------------: | :-----------: | :----------: | :--------------: | :---------: | :--------: | | **Knowledge & Reasoning** | | | | | | | | MMLU-Pro (EM) | 83.0 | 86.4 | 86.2 | 82.9 | 87.1 | **87.5** | | SimpleQA-Verified (Pass\@1) | 23.1 | 28.9 | 34.1 | 45.0 | 46.2 | **57.9** | | Chinese-SimpleQA (Pass\@1) | 71.5 | 73.2 | 78.9 | 75.8 | 77.7 | **84.4** | | GPQA Diamond (Pass\@1) | 71.2 | 87.4 | 88.1 | 72.9 | 89.1 | **90.1** | | HLE (Pass\@1) | 8.1 | 29.4 | 34.8 | 7.7 | 34.5 | **37.7** | | LiveCodeBench (Pass\@1) | 55.2 | 88.4 | 91.6 | 56.8 | 89.8 | **93.5** | | Codeforces (Rating) | - | 2816 | 3052 | - | 2919 | **3206** | | HMMT 2026 Feb (Pass\@1) | 40.8 | 91.9 | 94.8 | 31.7 | 94.0 | **95.2** | | IMOAnswerBench (Pass\@1) | 41.9 | 85.1 | 88.4 | 35.3 | 88.0 | **89.8** | | Apex (Pass\@1) | 1.0 | 19.1 | 33.0 | 0.4 | 27.4 | **38.3** | | Apex Shortlist (Pass\@1) | 9.3 | 72.1 | 85.7 | 9.2 | 85.5 | **90.2** | | **Long Context** | | | | | | | | MRCR 1M (MMR) | 37.5 | 76.9 | 78.7 | 44.7 | 83.3 | **83.5** | | CorpusQA 1M (ACC) | 15.5 | 59.3 | 60.5 | 35.6 | 56.5 | **62.0** | | **Agentic** | | | | | | | | Terminal Bench 2.0 (Acc) | 49.1 | 56.6 | 56.9 | 59.1 | 63.3 | **67.9** | | SWE Verified (Resolved) | 73.7 | 78.6 | 79.0 | 73.6 | 79.4 | **80.6** | | SWE Pro (Resolved) | 49.1 | 52.3 | 52.6 | 52.1 | 54.4 | **55.4** | | SWE Multilingual (Resolved) | 69.7 | 70.2 | 73.3 | 69.8 | 74.1 | **76.2** | | BrowseComp (Pass\@1) | - | 53.5 | 73.2 | - | 80.4 | **83.4** | | HLE w/ tools (Pass\@1) | - | 40.3 | 45.1 | - | 44.7 | **48.2** | | MCPAtlas (Pass\@1) | 64.0 | 67.4 | 69.0 | 69.4 | **74.2** | 73.6 | | GDPval-AA (Elo) | - | - | 1395 | - | - | **1554** | | Toolathlon (Pass\@1) | 40.7 | 43.5 | 47.8 | 46.3 | 49.0 | **51.8** | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://unsloth.ai/docs/models/deepseek-v4.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.