Karpathy's autoresearch lets AI agents run experiments overnight

autoresearch is a lean, open-source script by Andrej Karpathy that lets an AI agent run machine learning experiments overnight while you sleep.

The entire project is just 630 lines of code, published on GitHub under an MIT license. It is not an enterprise framework or a heavy research suite. It is a dead-simple loop designed to automate the scientific method without human intervention.

What it does

You hand autoresearch a standard training script and give it a strict compute budget per run, typically five minutes on a GPU. The agent takes it from there.

In Karpathy's initial overnight run, the agent executed 126 experiments by itself, driving validation loss from 0.9979 down to 0.9697.

He then let it loose on a 12-layer model for two full days. Over that span, the agent processed roughly 700 autonomous changes and surfaced about 20 additive improvements that translated seamlessly to larger setups.

Stacking those changes dropped the training time on the standard GPT-2 benchmark from 2.02 hours to 1.80 hours, delivering an 11% efficiency gain. Karpathy noted that the agent even caught subtleties in attention scaling and regularization that he had personally missed over two decades of manual tuning.

How it works

The setup functions as an autonomous optimization loop that operates straight on raw code.

First, the agent inspects its own training script. It forms a hypothesis, such as nudging a learning rate or adjusting architectural depth, and edits the code directly. Next, it runs a short training job and checks validation loss in bits per byte.

If the loss improves, the agent keeps the commit. If the run regresses or breaks, it reverts the change and tries a different hypothesis. Everything happens in an endless, local trial-and-error cycle.

Because the script is tiny, getting it running is straightforward. You only need a working Python setup, training hardware, and access to an LLM to power the agent logic. The code is freely available on GitHub.

How builders are expanding it

The simplicity of the script prompted immediate experiments across the community.

Varun Mathur, CEO of Hyperspace AI, adapted the single-agent loop across a peer-to-peer network. In one night, 35 autonomous nodes ran 333 experiments without supervision.

Hardware limitations produced distinct emergent behavior. H100 GPUs used brute force to tune learning rates, while CPU-bound laptops focused on initialization and normalization tweaks. By sharing wins over a gossip protocol, the network independently rediscovered milestones like RMSNorm in just 17 hours.

Marketers are testing the same loop outside of code. Eric Siu, founder of Single Grain, adapted the concept to marketing assets, swapping training scripts for landing pages and cold emails to optimize reply rates automatically around the clock.

Why it matters

We think the real brilliance of autoresearch is its restraint. By keeping the script at 630 lines, Karpathy turned machine learning tuning into a fast evolutionary process that anyone with a GPU can spin up in minutes. If you spend hours testing hyperparameters manually, this is worth running tonight.

That said, the approach has clear boundaries. It is fundamentally a local hill-climbing machine. It optimizes relentlessly against a single validation metric, which means it will polish existing architectures brilliantly, but it will not invent transformer-level breakthroughs from scratch.

Still, as an automated assistant for low-level experimentation, it changes the workflow. Handing off tedious trial-and-error to silicon lets developers spend their waking hours on higher-level system design.