Dashboard

What Is NVFP4? The 4-Bit Format Explained

NVFP4 is NVIDIA's 4-bit floating point format for AI models. Here is how the 16-value blocks and two-level scaling work, with a worked rounding example.

Carlo Zuercher
Carlo Zuercher
Staff Engineer, Platform
16 October 20261 min read

NVFP4 is a 4-bit floating point number format that NVIDIA introduced for running AI models on its Blackwell GPUs. Each weight is stored in 4 bits, and every group of 16 weights shares an 8-bit scale factor, so the real cost is 4.5 bits per value. That makes a model about 3.5 times smaller than in FP16 and about 1.8 times smaller than in FP8, according to NVIDIA's own description of the format.

It matters this week because two open releases lean on it: Reflection's 501B Beam model ships an NVFP4 checkpoint, and FLUX.3 Image Klein lists NVFP4 memory figures.

What the 4 bits actually are

NVFP4 uses the E2M1 layout: 1 sign bit, 2 exponent bits and 1 mantissa bit. With so few bits there are only a handful of values a weight can take. The magnitudes are 0, 0.5, 1, 1.5, 2, 3, 4 and 6, plus their negatives. Everything in the model has to be rounded to one of those.

That looks hopeless until you add the scale. A weight is stored as a small number from that grid, and the real value is the grid number multiplied by a scale factor. Change the scale and the same grid covers a different range.

Why blocks of 16 matter

NVFP4 applies the scale to a micro-block of 16 values. Each block gets its own scale, stored as an 8-bit FP8 E4M3 number, which can be a fractional value rather than only a power of two. On top of that there is a second-level FP32 scale for the whole tensor.

Smaller blocks follow local changes in the data more closely. NVIDIA contrasts this with MXFP4, which uses blocks of 32. The cost of the smaller block is storage: one 8-bit scale per 16 values adds half a bit per value. That is where 4.5 bits comes from.

A worked example

Take a block of 16 weights where the largest absolute value is 3.0. A simple choice of scale maps that largest value to the top of the grid, 6, so the scale is 3.0 / 6 = 0.5. This sketch assumes the scale itself is stored exactly, which real FP8 scales are not, but it shows the idea.

Original weight

Divided by scale 0.5

Nearest grid value

Stored back as

Error

0.2

0.4

0.5

0.25

0.05

0.9

1.8

2

1.0

0.10

2.6

5.2

6

3.0

0.40

3.0

6

6

3.0

0.00

Two lessons fall out. The grid is coarse at the high end, so large weights lose the most in absolute terms. And the scale is doing the heavy lifting: if one block had a single huge outlier, every other weight in that block would be squeezed into the bottom of the grid. Small blocks limit how far one outlier can spread damage, and that is the design reason for 16.

What it saves

For a 70 billion parameter model, weights alone come to:

Format

Bits per weight

Weight memory

FP16

16

140 GB

FP8

8

70 GB

NVFP4

4.5

about 39 GB

That is arithmetic on parameter counts, not a measurement, and it leaves out the KV cache and activations. The ratios match NVIDIA's 3.5x and 1.8x figures. For the general idea of shrinking weights, what quantization is in AI is the place to start, and how parameters turn into memory fills in the 70 billion figure.

What it costs in accuracy

NVIDIA reports the post-training quantization results for DeepSeek-R1-0528 comparing FP8 with NVFP4:

Benchmark

FP8

NVFP4

MMLU-PRO

85%

84%

GPQA Diamond

81%

80%

LiveCodeBench

77%

76%

AIME 2024

89%

91%

NVIDIA summarises this as 1% or less degradation on key language tasks. It is the vendor's own test on one model, so treat it as a reasonable expectation rather than a guarantee for yours.

Do you need to care about it?

If you call a hosted API, the provider chooses the format and you never see it. NVFP4 matters when you host weights yourself. It needs Blackwell hardware: NVIDIA's write-up attributes the format to Blackwell and does not mention it on Hopper or Ampere GPUs. On older cards, FP8 checkpoints are the usual alternative, which is why model pages such as Beam's recommend FP8 for H200s. If you are sizing a machine, how much VRAM a local LLM needs shows how to budget beyond the weights, and how AI models work gives the wider background.

NVFP4 FAQ

Is NVFP4 the same as FP4?

No. FP4 is the general idea of 4-bit floating point. NVFP4 is NVIDIA's specific version with 16-value blocks, FP8 block scales and a second-level FP32 scale.

Which GPUs support NVFP4?

NVIDIA's write-up ties it to the Blackwell architecture. Older GPUs would need a different format, such as FP8.

How much memory does NVFP4 save?

Roughly 3.5 times versus FP16 and 1.8 times versus FP8 for the weights, because the real storage is 4.5 bits per value.

Does NVFP4 hurt model quality?

NVIDIA's DeepSeek-R1-0528 results show about a point of difference or less against FP8 on the benchmarks it lists, with AIME 2024 slightly higher. Results vary by model.

How did this land?

About the author

Carlo Zuercher
Carlo Zuercher

Staff Engineer, Platform

Carlo works on the platform that turns prompts into running apps. He writes the engineering deep dives and the changelog notes worth reading.

Share

Get the next post in your inbox

One email a month. Product updates, engineering posts, and the best of Built with Swarmz.

I agree to receive emails about AI building tips and Swarmz product news. Unsubscribe any time.