Sakana AI's Transformer² rewires its weights in real time

Nature figured out dynamic adaptation millions of years ago. Octopuses shift skin pigment to vanish into reefs, and human brains rewire damaged pathways to restore broken functions. AI models, on the other hand, are rigid statues with weights frozen solid the moment pre-training ends.

Tokyo-based research lab Sakana AI wants to shake up that paradigm. The team just unveiled Transformer², a self-adaptive machine learning system that alters an LLM's internal weights in real time based on the prompt it receives.

Surgical precision and amplification dials

Every capability in a language model is trapped inside massive weight matrices. To make these matrices flexible without retraining the entire model, Sakana AI turned to Singular Value Decomposition, a classic mathematical method that breaks down complex matrices into independent principal components.

Think of it as dissecting the model's brain into distinct neural pathways, separating components responsible for math, coding, or common sense reasoning.

Once decomposed, the researchers introduced Singular Value Finetuning. Using reinforcement learning instead of rigid supervised training, the system learns ultra-compact vectors known as z-vectors. These vectors act like amplifier dials, turning up the volume on components vital to a specific task while turning down the noise from irrelevant ones. Because SVF uses reinforcement learning, it bypasses the need for perfectly labeled ground-truth answers.

The two-pass trick and hybrid thinking

To apply these adjustments on the fly, Transformer² uses a two-pass inference process. In the first pass, the model figures out what kind of task it is looking at. In the second pass, it dials in the appropriate z-vectors to adjust its weights before generating the final output.

The framework detects tasks using one of three strategies: a classifier prompt, a dedicated lightweight classifier, or a few-shot optimization algorithm that blends multiple vectors together.

Sakana AI tested the method on Llama and Mistral across math, coding, reasoning, and visual question answering benchmarks. In tests on datasets like GSM8K and MBPP-Pro, the method consistently outperformed standard LoRA fine-tuning while requiring a fraction of the parameter footprint.

One fascinating discovery emerged when the model tackled tough mathematical puzzles on the MATH benchmark. Instead of relying solely on its math-specific z-vectors, the few-shot system automatically blended math, programming, and logical reasoning vectors together. The researchers even took z-vectors trained on Llama and plugged them into Mistral, watching Mistral gain performance boosts across most benchmarks without retraining.

Why it matters

For developers building complex agentic systems, we think Transformer² offers a refreshing alternative to the current headache of juggling multiple LoRA adapters. Stacking and swapping specialized adapters often bloats system memory, but dynamically adjusting intrinsic singular values introduces almost negligible overhead while keeping the underlying model lightweight.

On an architectural level, this work challenges the brute-force philosophy of building ever-larger static models. Instead of inflating parameter counts to memorize every scenario, teaching a compact model to morph its internal brain on demand looks like a far more elegant path toward continuous learning.

The real catch lies in latency. Running a two-pass inference loop inevitably adds overhead, making it tough to swallow in real-time, ultra-fast streaming products. Furthermore, while the cross-model transfer worked between structurally similar architectures like Llama and Mistral, proving this flexibility across fundamentally different foundation models will be the true test of the concept.