Sakana AI unveils CycleQD to build multi-skilled models without fine-tuning

If you want an AI model that can code, query databases, and manage operating systems, the standard playbook is simple but brutal: throw all that data into one massive pool and burn a small fortune on fine-tuning.

Sakana AI thinks that approach is broken. The Tokyo-based research lab just introduced CycleQD, a new framework that uses evolutionary algorithms to breed multi-skilled language models without touching slow, expensive traditional fine-tuning.

Instead of chasing a single mega-model, Sakana wants to build swarms of diverse, specialized agents that evolve together.

Breeding better models with SVD and merging

CycleQD is based on Quality Diversity (QD), an evolutionary computing concept aimed at finding a diverse library of high-performing solutions.

The system starts with a few expert models that each excel at a single task. To combine them, it applies two core evolutionary moves: crossover and mutation.

Crossover relies on model merging. Instead of retraining neural networks from scratch, CycleQD blends the parameters of two parent models together. It is fast, cheap, and immediately creates a hybrid with blended capabilities.

Mutation introduces fresh behaviors using singular value decomposition (SVD). The algorithm factorizes weight matrices to isolate fundamental sub-skills, tweaks them, and lets the model explore new behaviors that neither parent had. This keeps the models from falling into repetitive patterns and guards against overfitting.

To keep models balanced, CycleQD rotates priorities across generations. In round one, coding might be the main benchmark while database and OS skills are tracked as secondary traits. In round two, the focus switches. Sakana says this ensures every skill gets its moment in the spotlight without being drowned out.

Beating brute-force fine-tuning

Sakana tested CycleQD using Llama 3-8B checkpoints trained on coding, database operations, and OS tasks.

The evolutionary approach beat both traditional model merging and conventional multi-task fine-tuning across the board.

Even more striking was the inefficiency of the old way: fine-tuning a baseline model on all datasets combined barely outperformed single-skill experts, despite consuming dramatically more compute and time. CycleQD, meanwhile, produced hundreds of viable variants with different skill profiles at a fraction of the cost.

Sakana researchers believe this framework paves the way for lifelong learning in AI, where models accumulate capabilities continuously rather than starting from scratch.

Why it matters

For developers and enterprise teams, CycleQD offers a practical way out of GPU bankruptcy. Fine-tuning an 8B model across conflicting domains is notoriously finicky. Being able to take off-the-shelf specialist weights and algorithmically blend them into custom multi-task agents changes the unit economics of deploying local models.

For the industry, it is another vote against the monolithic scaling religion. While hyperscalers keep pouring billions into training trillion-parameter generalists, Sakana is showing that populations of smaller, specialized models might be a much more sustainable architecture.

We think the use of SVD for controlled mutation is brilliant—it treats neural weights like modular building blocks rather than a mysterious black box. That said, evolutionary algorithms are notoriously prone to plateauing as task complexity scales up. Merging weights works remarkably well for adjacent domains like code and command line tools, but whether this holds up when bridging vastly different cognitive skills remains to be seen.