🦥Run Unsloth Dynamic NVFP4 Guide

Unsloth Dynamic NVFP4 is a quantized model format that runs on NVIDIA Blackwell GPUs and is designed for faster, more accurate 4-bit inference. It combines NVIDIA’s native NVFP4 precision with Unsloth Dynamic 2.0 quantization to preserve model accuracy while reducing VRAM usage and increasing speed. This guide explains FP4 quantization, compares NVFP4 with other formats, and shows how to run models like Qwen3.8, Gemma 4 and Qwen3.6 locally using vLLM or SGLang on RTX 5050-5090, B200, RTX PRO 6000 and more GPUs.

Dynamic NVFP4 works by selecting important layers to remain in FP8 (W8A8) or BF16 and the rest in W4A4 (not W4A16) instead of forcing every layer into FP4. This allows up to 2.5x faster inference since W4A4 leverages Blackwell GPU's FP4 tensor cores. For all quants, we also provide FP8 KV cache calibration allowing for 2x longer context lengths.

Float4 vs other precisions

The trick for faster GPUs is to lower the numerical precision of matrix multiplications. The number of transistors needed for the matrix multiplication units is related to the square of the mantissa. The mantissa allows for numbers to have how many "fractional" decimals - so the more bits, the more accurate it can represent decimals. For example expressing 0.121332 is possible with more mantissa bits, whilst few mantissa bits will round it to 0.1.

FP32 has 23 mantissa bits, so 23^2+ 8 exponent bits = 537 space is needed. Bfloat16 has 7 mantissa bits, so 7^2 + 8 exponent = 57 space. This means bfloat16 needs around 9x less space than FP32! And when we go to float8 which has 3 mantissa bits so 3^2 + 4 exponent = 13 - this is 41x less space than FP32!

Finally float4 has 1 mantissa bit and 2 exponents so 3 space - a whopping 179x less space than FP32 - this essentially means a GPU can do around 179x more FP4 matrix multiplication than FP32 multiplication FLOPs in the same space!

NVFP4 vs MXFP4

There is another FP4 format called MXFP4 - it's less accurate than NVFP4 due to 2 things:

1. NVFP4 uses a block size of 16 vs 32 for MXFP4 - this allows outliers to be isolated easier and scaling factors are provided for smaller subsets of weights which increases accuracy

2. A E4M3 (FP8) scale is used instead of a E8M0 (powers of 2 scaling) per block. Using a FP8 type block size looks to be much better especially for LLMs.

Performance Analysis

Our new dynamic NVFP4 Qwen3.6 quants run ~2.5× faster than other NVFP4 quants, with better performance and comparable file sizes. Run Qwen3.6-27B NVFP4 2.5x faster on 24GB VRAM and Qwen3.6-35B-A3B 1.7x faster on 32GB VRAM. We also added FP8 KV cache calibration for 2x longer context lengths! NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see Unsloth Dynamic NVFP4), B200, B300 GPUs. For older GPUs, our GGUFs work well!

All benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s. We also just released our new Qwen3.8 NVFP4 quants:

• Qwen3.8-27B NVFP4 (new)

We're also releasing two 35B-A3B NVFP4 versions:

• Qwen3.6-35B-A3B-NVFP4-Fast which is a full W4A4 quant - 1.79x faster

• Qwen3.6-35B-A3B-NVFP4 which is slightly bigger but more accurate and 1.56x faster

For accuracy benchmarks, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:

MTP tensors are also built directly into the quants for additional speedups. Accuracy gains come from improvements to Qwen3.6’s chat template and dataset calibration. We use our previous chat template updates to help improve coding and tool-calling consistency while reducing looping and other reported issues. Our calibration uses a mix of our dataset optimized for coding, tool-calling and chat alongside UltraChat.

For Decode speed (tokens per person), ours is 1.03x faster for 27B and 1.17x and 1.22x faster for 35B.

Overview

Below are the hardware requirements for models which you can use including Gemma 4 and Qwen3.6. Also see the overall speed boost you will achieve:

Gemma 4:

Qwen3.6:

NVFP4 Benchmarks

NVFP4 runs 4-bit weights and matrix multiplications directly on Blackwell Tensor Cores. Our Qwen3.6 NVFP4 quants use W4A4 so they actually use the FP4 tensor cores, so they decode faster than NVIDIA's which use W4A16. We also dynamically quantize layers to retain accuracy, and we conducted MMLU-Pro, AIME 2025, GPQA for all quants including comparing to FP8 and BF16.

Qwen3.6-27B NVFP4 Accuracy Benchmarks

Qwen3.6-35B-A3B NVFP4 Accuracy Benchmarks

We also checked the output length of all benchmarks, and they are comparable, so the new NVFP4 quants do not think for longer which defeats the purpose of quantizing them! (Ie if it's 2x faster, but thinks 2x more, then that's useless)

Run NVFP4 Tutorials

To run NVFP4 quants, see below for commands to run Qwen3.6-27B in vLLM and SGLang (you can change model name to Qwen3.6-35-A3B-NVFP4).

vLLM Tutorial

You can run all NVFP4 models in vLLM. Do NOT select any MoE backend - leave vLLM to select it - for eg Marlin is 2.5x slower! See Marlin vs Flashinfer vs cutlass vs cute-DSLIf you have a DGX Spark, see Unsloth Dynamic NVFP4 you must use --moe-backend flashinfer_b12x or you will get much slower inference.

To install vLLM in a separate venv:

Then to serve the 35B Fast variant:

Change unsloth/Qwen3.6-35B-A3B-NVFP4-Fast to the NVFP4 quant names!

To enable MTP / speculative decoding (faster decode but somewhat less throughput), use:

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

DGX Spark Tutorial

To ensure DGX Spark has the correct kernels (or you will get 2x SLOWER inference), first check:

which should NOT error out - if it did, please update vllm or reinstall via:

Then to serve in vLLM for DGX Spark:

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

SGLang Tutorial:

You can run all NVFP4 models in SGLang. Remember to switch out the model name for your desired model.

Qwen3.6:

Gemma 4:

Gemma 4 and others

Every Gemma 4 variant now has an Unsloth Dynamic NVFP4 checkpoint.

We show Gemma-4 having at most a 1.44x throughput boost on serving 128 concurrent people on 1x B200 vs BF16. Qwen3.5-122B-A10B is 1.38x faster and GLM-4.7-Flash is 1.27x faster.

Marlin vs Flashinfer vs cutlass vs cute-DSL

We also found Marlin kernels to not support W4A4 well - enabling it will cause a 2.5x performance degradation - so use CUTLASS, Flashinfer-TRTLLM or Cute-DSL (auto enabled in vLLM)! Also if you have a DGX Spark, see Unsloth Dynamic NVFP4 you must use --moe-backend flashinfer_b12x or you will get 2.5x slower inference.

So don't set any backend - vLLM auto selects the best.