𝗡𝗩𝗜𝗗𝗜𝗔 𝗕𝗹𝗮𝗰𝗸𝘄𝗲𝗹𝗹 𝗦𝗽𝗲𝗲𝗱𝘀 𝗨𝗽 𝗝𝗔𝗫 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴

𝗡𝗩𝗜𝗗𝗜𝗔 𝗕𝗹𝗮𝗰𝗸𝘄𝗲𝗹𝗹 𝗦𝗽𝗲𝗲𝗱𝘀 𝗨𝗽 𝗝𝗔𝗫 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴

NVIDIA released NVFP4. This 4-bit format runs on Blackwell GPUs. It speeds up JAX training in MaxText.

You get 1.8x faster training over FP8. The format puts two 4-bit values in one 8-bit register. This doubles the math density.

Here are the key facts:

• 1.8x training speedup over FP8.
• No accuracy loss for models up to 70B parameters.
• Native support in Google MaxText.
• Dedicated FP4 tensor cores in Blackwell hardware.

Blackwell GPUs save memory. They use half the memory of FP16. They use 1.5x less memory than FP8. This lets you use larger batch sizes.

The H100 lacks these FP4 cores. This gives Blackwell a clear edge for pre-training.

NVIDIA says accuracy stays high. We need more tests for models over 100B parameters. We need to see real scores to be sure.

Source: https://dev.to/gentic_news/nvidia-nvfp4-on-blackwell-cuts-jax-training-by-18x-in-maxtext-373a Optional learning community: https://t.me/GyaanSetuAi