Karpathy's $100 full-stack ChatGPT clone
Andrej Karpathy just dropped nanochat, an 8,000-line codebase that packs an entire end-to-end ChatGPT training and inference pipeline into a single repo. Running a single script on an 8xH100 node costs about $100 and yields a working, chat-ready model in just four hours, covering everything from tokenizer training to GRPO reinforcement learning. Why it matters: It turns the entire modern LLM post-training playbook into a readable, easily forkable playground for developers.
nanochat is Andrej Karpathy's new open-source pipeline that builds a working ChatGPT clone from scratch in a single, dependency-minimal repo. Clocking in at around 8,000 lines of clean code, it covers everything from bare-metal training to a working browser interface.
You spin up a cloud GPU instance, run a single script, and roughly four hours later you are chatting with your own language model inside a ChatGPT-like web UI.
What it does
While Karpathy's earlier nanoGPT only tackled pretraining, nanochat packages the complete modern post-training stack into one readable codebase.
The pipeline starts by training a custom tokenizer with a new Rust implementation. It pretrains a Transformer architecture on FineWeb, tracking progress with the CORE evaluation metric.
From there, it handles midtraining using conversations from SmolTalk, multiple-choice datasets, and tool-use examples. It then runs SFT supervised fine-tuning, benchmarked against MMLU, ARC, GSM8K, and HumanEval.
It even includes an optional reinforcement learning stage using GRPO on GSM8K math problems. The built-in inference engine handles KV caching, prefill/decode logic, and tool use via a lightweight Python sandbox, finishing with a gamified markdown report card.
Specs and cost
The baseline setup is remarkably accessible. Running an 8xH100 node for about four hours costs roughly $100 and yields a toy model capable of answering simple questions and writing short stories.
Pushing the run to around 12 hours pushes its CORE score past GPT-2. Scaling the budget toward $1,000 (about 41.6 hours) produces a significantly more coherent model capable of tackling simple math and coding tasks.
For example, training a depth-30 model for 24 hours matches the compute budget of GPT-3 Small (125M parameters), landing scores in the 40s on MMLU, 70s on ARC-Easy, and 20s on GSM8K.
Karpathy noted the codebase is intentionally unoptimized with plenty of low-hanging fruit left for the community. It is slated to serve as the capstone project for his upcoming LLM101n course, with code and speedrun instructions live on GitHub.
Why it matters
If you are an engineer or researcher trying to demystify modern LLM post-training, nanochat cuts through massive framework bloat. Instead of juggling multiple fragmented libraries for SFT, RL, and serving, you get an end-to-end blueprint you can actually read in an afternoon.
It is not designed to replace high-throughput production trainers, but as a hackable research harness and learning vehicle, it sets a new baseline. For anyone with a few hundred dollars in GPU credits who wants to experiment with reinforcement learning and tool calling from first principles, this is where you start.
Sources (1)
- @karpathy(opens in a new tab)Originalx.com