Moonshot AI rethinks Transformer residual connections with AttnRes
Moonshot AI just introduced Attention Residuals (AttnRes), a drop-in replacement that swaps standard fixed residual connections for dynamic attention across depth. This lets layers selectively pull from earlier representations without suffering from representation dilution. Its practical block variant matches a baseline trained with 1.25x more compute, delivering sharp jumps in complex reasoning and coding benchmarks.