Nvidia and Sakana make LLMs 30% faster by doing less math

The human brain is remarkably efficient because it only fires up the specific neurons it actually needs. Modern large language models try to do the exact same thing: in standard feedforward layers, more than 95% of neurons stay silent for any given token.

Yet deep learning has long suffered from an annoying paradox: telling a model to do less math usually makes it run slower. That is because GPUs crave massive, predictable, dense blocks of numbers. Unstructured sparsity turns memory access into a messy scavenger hunt, and the scheduling overhead ruins performance.

Tokyo-based Sakana AI and Nvidia unveiled a clever way out. Instead of begging GPUs to handle messy sparsity, they reshaped the sparsity to look like dense blocks that Nvidia silicon loves.

Reshaping math for the GPU

The breakthrough centers on TwELL, short for Tile-wise ELLPACK, alongside a suite of custom-fused CUDA kernels.

Sakana and Nvidia designed a hybrid layout that slices activations into tidy hardware tiles. The system dynamically routes 99% of sparse tokens through a hyper-fast pathway, skipping unused computations without breaking the GPU tile-matrix rhythm. For the rare, dense tokens that cannot be easily skipped, a dense matrix acts as a safety valve.

By matching the data structure to existing hardware pipelines, the team eliminated the memory penalty that typically kills sparse algorithms.

More speed, less power, zero quality loss

The benchmark numbers on Nvidia H100 GPUs are impressive.

Tested on models at billion-parameter scales, the TwELL architecture boosted inference speed by over 30% while slashing energy consumption by more than 30%.

Training saw big gains as well: training speed increased by over 20%, while peak memory usage dropped by more than 20%. Crucially, downstream evaluation accuracy remained virtually untouched. Sakana and Nvidia have open-sourced the kernels, code, and formats ahead of presenting the paper at ICML 2026.

Why it matters

We think this collaboration solves one of the most frustrating bottlenecks in AI infrastructure.

For companies running large fleets of GPUs, power bills and memory bandwidth are the true limiting factors. Typically, cutting serving costs meant shrinking the model or quantizing weights, which almost always comes with a small tax on output quality. Proving that real hardware speedups can be extracted from algorithmic sparsity without sacrificing model capability is a huge win for anyone operating at scale.

That said, the approach is not an instant silver bullet for every setup. The current benchmarks were demonstrated at billion-parameter scales on H100s. Scaling TwELL across multi-node tensor parallel clusters could introduce new communication quirks if token sparsity varies too wildly across GPUs.

Still, seeing Nvidia build dedicated CUDA plumbing around dynamic sparsity marks a critical shift. The easiest way to get 30% more out of existing silicon might not be waiting for the next chip generation, but simply teaching GPUs how to skip the math they never needed to do.