DeepSeek-V4: How to Run Locally
Run DeepSeek-V4-Flash locally on your own device!
DeepSeek-V4 is DeepSeek's new open models including DeepSeek-V4-Pro with 1.6T parameters (49B active) and DeepSeek-V4-Flash with 284B parameters (13B active). The models excel at coding, agentic workflows and chat with a 1M context window. In this guide we'll show you how to run DeepSeek-V4-Flash locally: DeepSeek-V4-Flash GGUF
For lossless DeepSeek, use Q8 (UD-Q8_K_XL), which is only 7GB larger than Q4 (UD-Q4_K_XL). The lossless 8-bit GGUF is 162 GB and 3-bit is 103GB which can run on a 110GB RAM device. DeepSeek-V4-Flash scores 86.2% on MMLU-Pro, 88.1% on GPQA Diamond, and 56.9% on Terminal Bench 2.0.
July 7: DeepSeek-V4 is now ready to run! We also improved the DeepSeek-V4 chat jinja template, and tested over 4000 conversations to be equivalent with the official baseline.
🦙 llama.cpp DeepSeek V4 implementation fixes
llama.cpp added DeepSeek V4 support in 24162 - we noticed that when using any GGUF from any provider, multi turn conversations would not function well when compared to DS4's Hugging Face baseline when using KV cache quantization --cache-type-k/v q8_0 - it has since been fixed and merged in llama.cpp since July 7th 2026 with 25202
Specifically before the fix, calling DeepSeek quants would cause overlayotin kinetic academyléléléléulif and after the PR "The capital of France is Paris." which is correct.
💬 DeepSeek V4 Chat template improvements
We also improved the DeepSeek-V4 chat jinja template, and tested over 4000 conversations to be equivalent with the golden baseline (official DS4)
We added reasoning_effort and you can select max, high just like official DeepSeek-V4. We prepend the correct system prompt as per DS4, and followed gpt-oss's style.
And for tool calls, reasoning_content was retained for DS4, but the jinja chat template would exclude them. We added it back.
Disabling Thinking, changing reasoning effort
DeepSeek-V4 uses reasoning by default. It also supports reasoning efforts where reasoning_effort can be "high", "max" or disabled.
To disable thinking, use --chat-template-kwargs '{"enable_thinking":false}'. If you're on Windows Powershell, use: --chat-template-kwargs "{\"enable_thinking\":false}"
You can also use --reasoning on or --reasoning off in llama.cpp as well now!
For reasoning effort customization and or to disable reasoning, use the below examples:
📊 Quantization Analysis
Our UD-Q8_K_XL quant is fully lossless. DeepSeek-V4-Flash is quantization-aware-trained: the official checkpoint stores its routed experts (96% of the model) natively in MXFP4 and everything else in FP8 or BF16. GGUF's MXFP4 is exactly that format, so we repack the experts bit-for-bit, and FP8 dequantizes into BF16 with no rounding. We checked every tensor against the official DeepSeek weights: all 1,328 are bit-identical, and it stays lossless at inference (KL-divergence ~0, 100% top-token agreement).
Non-Unsloth DeepSeek-V4-Flash GGUFs were converted without these paths thus deviating from the official weights. UD-Q4_K_XL keeps the same bit-exact experts and only quantizes the non-expert tensors (4% of the model) to Q8_0, so it sits right next to Q8 in size and quality.
Measured against the official weights, both Unsloth quants are on the quality/size frontier. UD-Q8_K_XL is the only lossless point. UD-Q4_K_XL matches other community MXFP4 formats and is more accurate than the Q4_K-experts conversions, which are larger yet land at 0.029 KLD.
The error split by layer shows why. Keeping the native MXFP4 experts means 0% weight error at every layer. Conversions that re-quantize the experts to Q4_K or IQ2_XXS round almost every weight: 5% for Q4_K, over 30% for IQ2_XXS.
We also found using Q8_0 and F16 for some tensors is not lossless, and it gets worse since QAT was applied by DeepSeek to make MXFP4 / FP8 work well, so we had to leave them in BF16 directly. So use UD-Q8_K_XL for a true lossless quant, and UD-Q4_K_XL downcasts some of the BF16 items to Q8_0.
For full benchmark tables of GGUF Benchmarks, see here.
⚙️ Usage Guide
DeepSeek-V4-Flash is smaller and faster than DeepSeek-V4-Pro, with 284B parameters (13B active), and a 1M context window. The model has 3 modes, Non-think, Think High and Think Max.
It's recommended to use UD-IQ3_XXS which is 103GB for best results. Because the file size does not include KV cache, context allocation, try to have at least 110GB RAM to run the model.
The UD-Q8_K_XL quant is DeepSeek-V4-Flash in full original precision. It is 162GB size and it's best to have at least 169GB of available RAM/VRAM available.
Table: Inference hardware requirements (units = total memory: RAM + VRAM, or unified memory)
92 GB
102 GB
110-135 GB
162 GB
169 GB
For best performance, make sure your total available memory, including VRAM and system RAM, exceeds the quantized model file size by a comfortable margin.
Recommended Settings
DeepSeek recommends these parameters for best performance: temperature=1.0, top-p=1.0
Think High is on by default. If disabled, you can enable it via: --chat-template-kwargs '{"enable_thinking":true}' or toggle it via the UI dropdown in Unsloth Studio. Also see DeepSeek V4 Chat template improvements
temperature = 1.0
top-p = 1.0
Maximum context window:
1,048,576For Think Max, set context to at least 384K tokens.
Run DeepSeek-V4-Flash Tutorials:
For this tutorial, we will use the 3-bit quant UD-IQ3_XXS, as it fits on a 128GB RAM device. Replace UD-IQ3_XXS with UD-Q8_K_XL (original quality) or another quant if your machine has enough memory. You can now run DeepSeek-V4-Flash in Unsloth Studio.
🦥 Unsloth Studio Guide🦙 Llama.cpp Guide
🦥 Unsloth Studio Guide
DeepSeek-V4-Flash can now be run and trained in Unsloth Studio, our new open-source web UI for local AI. Unsloth Studio lets you run models locally on MacOS, Windows, Linux and:
Search, download, run GGUFs and safetensor models
Self-healing tool calling + web search
Code execution (Python, Bash)
Automatic inference parameter tuning (temp, top-p, etc.)
Fast CPU + GPU inference via llama.cpp
Train LLMs 2x faster with 70% less VRAM
Search and download DeepSeek-V4-Flash
On first launch you will need to create a password to secure your account and sign in again. Then go to the Studio Chat tab and search for DeepSeek-V4-Flash in the search bar and download your desired model and quant.
Run DeepSeek-V4-Flash
Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. Because Think High is on by default, you can go to the right dropdown to toggle it to Non-think or Think Max. You can also edit the context length, chat template and other settings.
For more information, you can view our Unsloth Studio inference guide.
🦙 Llama.cpp Guide
Obtain the latest llama.cpp on GitHub here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default.
You can now use llama.cpp directly to load and download models, just like ollama run. First, select the quantization type you want like IQ3_XXS. Also use export LLAMA_CACHE="folder" to force llama.cpp to save to a specific location. Note this download process might be very slow, so it's probably best to use the manual download process in the next section.
If you want to download the model manually, we can download the model via the code below (after installing pip install huggingface_hub). If downloads get stuck, see: Hugging Face Hub, XET debugging
You can edit --threads 32 for the number of CPU threads, --ctx-size 32768 for context length, --n-gpu-layers 2 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
📊 Benchmarks
GGUF Benchmarks
See below for a table comparing benchmarks for quants from Unsloth and other providers. Reference = official weights. Perplexity and KL-divergence over wikitext-2 at ctx 512 on 4x B200.
Official (reference)
156.4
4.5319
0
0%
100%
100%
Unsloth UD-Q8_K_XL
161.9
4.5319
~0 (lossless)
0.000%
100.000%
100.000%
Unsloth UD-Q4_K_XL
155.1
4.5335
0.0102
3.40%
96.28%
97.46%
bartowski MXFP4
156.0
4.5351
0.0105
3.42%
96.18%
97.57%
antirez Q4KExperts-F16 (imatrix)
164.6
4.5743
0.0291
5.87%
93.95%
0.51%
antirez Q4KExperts-F16
164.6
4.5726
0.0290
5.89%
93.94%
0.93%
antirez mixed L37-42-Q4K (imatrix)
97.6
5.8169
0.3605
21.15%
79.74%
0.41%
antirez IQ2XXS (imatrix)
86.7
6.0808
0.4079
22.23%
78.15%
0.39%
antirez IQ2XXS
86.7
6.1518
0.4207
22.74%
77.92%
0.47%
Official Benchmarks
See further below for benchmarks in table format:
Knowledge & Reasoning
MMLU-Pro (EM)
83.0
86.4
86.2
82.9
87.1
87.5
SimpleQA-Verified (Pass@1)
23.1
28.9
34.1
45.0
46.2
57.9
Chinese-SimpleQA (Pass@1)
71.5
73.2
78.9
75.8
77.7
84.4
GPQA Diamond (Pass@1)
71.2
87.4
88.1
72.9
89.1
90.1
HLE (Pass@1)
8.1
29.4
34.8
7.7
34.5
37.7
LiveCodeBench (Pass@1)
55.2
88.4
91.6
56.8
89.8
93.5
Codeforces (Rating)
-
2816
3052
-
2919
3206
HMMT 2026 Feb (Pass@1)
40.8
91.9
94.8
31.7
94.0
95.2
IMOAnswerBench (Pass@1)
41.9
85.1
88.4
35.3
88.0
89.8
Apex (Pass@1)
1.0
19.1
33.0
0.4
27.4
38.3
Apex Shortlist (Pass@1)
9.3
72.1
85.7
9.2
85.5
90.2
Long Context
MRCR 1M (MMR)
37.5
76.9
78.7
44.7
83.3
83.5
CorpusQA 1M (ACC)
15.5
59.3
60.5
35.6
56.5
62.0
Agentic
Terminal Bench 2.0 (Acc)
49.1
56.6
56.9
59.1
63.3
67.9
SWE Verified (Resolved)
73.7
78.6
79.0
73.6
79.4
80.6
SWE Pro (Resolved)
49.1
52.3
52.6
52.1
54.4
55.4
SWE Multilingual (Resolved)
69.7
70.2
73.3
69.8
74.1
76.2
BrowseComp (Pass@1)
-
53.5
73.2
-
80.4
83.4
HLE w/ tools (Pass@1)
-
40.3
45.1
-
44.7
48.2
MCPAtlas (Pass@1)
64.0
67.4
69.0
69.4
74.2
73.6
GDPval-AA (Elo)
-
-
1395
-
-
1554
Toolathlon (Pass@1)
40.7
43.5
47.8
46.3
49.0
51.8
Last updated
Was this helpful?