NVIDIA TensorRT cuts Stable Diffusion 3.5 VRAM by 40% and doubles speed

Running the best open image models locally usually hits a hard wall: VRAM.

Stable Diffusion 3.5 Large natively demands over 18GB of video memory, effectively shutting out most everyday graphics cards. NVIDIA just teamed up with Stability AI to overhaul how the model runs on consumer hardware, using TensorRT to slash both hardware requirements and generation times.

By quantizing SD3.5 Large down to FP8 precision, VRAM consumption drops by 40% to 11GB. That shift means five GeForce RTX 50 Series GPUs can now run the heavy model entirely inside onboard memory, rather than just the single highest-end flagship.

Faster generations, simpler developer plumbing

Speed gets a massive lift alongside the memory diet.

On SD3.5 Large, the combination of FP8 and TensorRT delivers a 2.3x performance boost compared to running the original BF16 model in standard PyTorch. The smaller SD3.5 Medium also gets love, showing a 1.7x speedup in BF16.

The optimized weights are already live on Stability AI's Hugging Face hub. NVIDIA and Stability also plan to launch an SD3.5 NIM microservice in July for easier API-style deployments.

Alongside the models, NVIDIA launched TensorRT for RTX as a standalone SDK. Previously, developers using TensorRT had to pre-compile separate engine binaries for every individual GPU model—a tedious and time-consuming headache. The updated version uses just-in-time compilation to build device-specific engines in seconds during install or first launch, and the SDK footprint is now 8x smaller. It is also integrated into Microsoft's new Windows ML inference preview.

Why it matters

Squeezing a flagship 18GB model down to 11GB brings top-tier local generation to mainstream creators. Instead of needing an expensive 24GB workstation card, users on mid-range 12GB and 16GB cards can now run SD3.5 Large locally at more than double the original speed.

We think the bigger play here is fixing TensorRT's developer ergonomics. TensorRT has always been fast, but packaging nightmare engines kept desktop app builders away. With an 8x smaller footprint and zero-effort JIT builds baked into Windows ML, NVIDIA is turning its proprietary acceleration into Windows' default AI engine. Rivals might match raw silicon numbers, but this kind of software trench-digging makes leaving the RTX ecosystem harder than ever.