SubQ brings 12M-token context with sparse attention

Shoving millions of words into a standard transformer model usually melts your GPU and your budget with it.

Subquadratic just unveiled SubQ, a new frontier-scale large language model designed to tear down that wall using a fully sub-quadratic sparse-attention architecture, or SSA.

Skipping the junk tokens

Standard transformers waste massive amounts of compute calculating relationships between every single word, even when most of those connections do not matter.

SubQ flips this by filtering for only the connections that actually carry meaning. According to the company, this cuts compute by nearly 1,000 times.

That architectural shift allows SubQ to boast a 12 million token context window. At the 1 million token mark, Subquadratic claims it runs 52 times faster than FlashAttention while costing less than 5% of Opus.

Why it matters

A 12 million token context window means feeding an entire enterprise codebase or hundreds of technical manuals straight into the prompt without building messy chunking pipelines.

That said, sparse-attention designs have historically struggled with needle-in-a-haystack recall and complex multi-step reasoning, where skipping relationships can lead to missed context.

In our view, claiming a 52x speedup and rock-bottom costs is an impressive shot across the bow of frontier labs. But until external developers run thorough benchmarks on its accuracy, treat those massive context claims as a fascinating architecture test rather than a done deal.