Manifest AI drops attention from Qwen3 for just $4,000

Ever since Google published Attention Is All You Need back in 2017, the transformer's core mechanism has reigned supreme. But as context windows stretch across massive codebases and long documents, attention's quadratic compute cost is hitting a hard wall.

Now, a little-known startup called Manifest AI wants to prove we can finally move on. The team just unveiled Brumby-14B-Base, a retrained variant of Qwen3-14B that strips out attention layers entirely.

In their place, the model runs on an alternative architecture dubbed Power Retention.

Trading attention for a memory matrix

Traditional transformers work by comparing every single token against every other token in the sequence. That pairwise comparison gives models their reasoning power, but it also causes memory and compute to explode as context grows.

Power Retention takes a very different path. While it keeps the familiar queries, keys, and values, it replaces global similarity matching with a recurrent state update.

Each layer updates a compact internal memory matrix at every step using learned gating. Because it acts more like an RNN than a standard transformer, processing a token costs the exact same amount of compute whether context is 1,000 tokens or a million tokens long.

To keep the expressive power of attention, the architecture uses tensor powers of the inputs. Manifest AI claims this retains long-range dependencies without the memory penalties.

The $4,000 catch

What really caught attention is the training budget. Manifest AI trained Brumby-14B on just 32 Nvidia H100 GPUs for 60 hours, ringing up a total bill of roughly $4,000.

Within 3,000 steps, Brumby converged to match the baseline training loss of Qwen3. On benchmark suites covering reasoning and comprehension, it performed on par with Qwen3-14B and GLM-4.5-Air.

That said, don't expect the transformer era to end tomorrow.

Manifest AI founder Jacob Buckman clarified that the tiny $4,000 price tag is only possible because they bootstrapped the model using existing Qwen3 transformer weights. Starting from scratch at that price would be impossible.

Still, Buckman argued that piggybacking on previous generation weights is a crucial shortcut. It proves that alternative architectures can catch up to frontier performance for orders of magnitude less money.

Why it matters

For developers wrestling with runaway inference costs on long-context tasks, Brumby offers a practical glimpse of constant-time token generation. If inference runtimes optimize for it, serving book-length prompts could become dramatically cheaper.

For the broader AI ecosystem, this rewrites the playbook on architectural innovation. Startups usually avoid ditching transformers because pretraining from scratch requires millions of dollars in compute. Manifest AI showed that you can perform open-heart surgery on an existing open-source giant instead.

We think this is one of the cleverest efficiency hacks of the year, but with a clear caveat. Because Brumby relies on Qwen3's pre-existing representations, it has not yet proven it can stand entirely on its own from step zero. Whether its recurrent memory truly holds up against transformer attention under messy real-world pressure remains the real test.