Moonshot AI rethinks Transformer residual connections with AttnRes

Standard residual connections are the unsung heroes of modern deep learning. Ever since ResNet showed us how to simply add inputs to outputs, that tiny addition has prevented gradient vanishing and paved the way for every massive Transformer we run today.

Yet, that decade-old trick has always had a blind spot. Standard residual streams treat every layer equally. As models scale deeper, accumulating features with fixed unit weights dilutes earlier representations and causes hidden state magnitudes to spiral out of control.

Moonshot AI and the Kimi team have now open-sourced an answer called Attention Residuals (AttnRes). It is a drop-in architectural rethink that ditches rigid additions in favor of dynamic softmax attention across depth.

Giving every layer selective hindsight

Traditional residual accumulation works like stacking paper: by layer 60, whatever you wrote on layer 3 is deeply buried under uniform noise. You cannot easily pull an early feature forward without dragging everything else along with it.

AttnRes changes the rule. Instead of adding outputs together, each layer uses a single learned pseudo-query vector to compute an attention distribution over all preceding layer outputs. If a deeper layer needs low-level syntactic primitives from layer 2, it simply shifts its attention weights there.

Doing this across every single layer can bloat memory to O(Ld). To make it practical for production training, the researchers designed Block AttnRes.

Block AttnRes divides the network into a small set of chunks, typically around 8 blocks. Inside each block, layers accumulate features via fast, traditional residuals. Attention is reserved exclusively for jumping across block boundaries. This brings memory down to O(Nd) with virtually negligible overhead, while keeping nearly all the quality gains.

Big reasoning gains on a smaller compute budget

Across evaluation suites, the numbers are hard to ignore. The researchers found that Block AttnRes matches the training loss of a baseline model that chewed through 1.25x more compute.

The biggest performance spikes landed squarely on multi-step reasoning and software engineering. On the GPQA-Diamond benchmark, AttnRes jumped from 36.9 to 44.4—a massive 7.5-point boost. HumanEval coding accuracy climbed from 59.1 to 62.2, while MBPP rose from 72.0 to 73.9.

General benchmarks followed suit: MMLU crept up from 73.5 to 74.6, BBH rose to 78.0, and Chinese evaluations like C-Eval advanced from 79.6 to 82.5. Crucially, the internal mechanics held up: output magnitudes remained cleanly bounded and gradient norms spread out far more evenly across depth.

Why it matters

We think this work is exceptionally clever because it targets a corner of deep learning that most labs took for granted. While everyone spent the last two years obsessing over horizontal context length and attention sparsification, Moonshot went vertical, proving the backbone that binds layers together still has plenty of juice to give.

For teams pre-training their own foundation models, getting a 1.25x compute multiplier almost for free is an enticing trade. The sharp improvement in GPQA-Diamond suggests that letting deeper reasoning circuits reach back to uncorrupted early signals is vital for multi-step logic.

The real catch will be scaling. While Block AttnRes keeps local memory usage tidy, wiring cross-layer attention into heavy pipeline-parallel clusters will inevitably complicate communication boundaries between GPUs. If engineering teams can smooth out those pipeline handshakes, standard residual additions might finally start retiring.