Nvidia and Sakana make LLMs 30% faster by doing less math
Sakana AI and Nvidia just tackled a notorious hardware bottleneck with TwELL, a new sparse packing format and custom CUDA kernel that stops GPUs from wasting math on inactive neurons. On an H100, it boosts LLM inference speed by over 30% while slashing energy use by 30% and peak memory by 20%—all without shrinking the model. Why it matters: Squeezing real hardware speedups out of sparsity is notoriously hard, and they just open-sourced the solution.