Andrej Karpathy drops a 2-hour masterclass on building GPT tokenizers

Just a week after leaving OpenAI, Andrej Karpathy is already back to doing what the AI community loves most: teaching. On February 20, he dropped a comprehensive two-hour deep dive on YouTube, showing how to build a GPT tokenizer completely from scratch.

Why LLMs struggle with basic spelling

Most users assume that when an LLM fails at simple math or can't reverse a word, the reasoning engine is simply flawed. Towards the end of the video, Karpathy breaks down the quirks of tokenization to prove that the root cause is often the tokenizer itself.

Because tokenizers slice text into arbitrary chunks rather than clean human concepts, numbers and letter sequences get scrambled before the model even starts thinking. He also dropped a neat practical tip for developers: feeding data to LLMs in YAML instead of JSON can save serious context, because YAML typically encodes the exact same data into fewer tokens.

Why it matters

If you build with LLMs or care about prompt engineering, this two-hour masterclass is well worth your afternoon. We think developers often fixate on model sizes and fine-tuning while ignoring the tokenizer, which acts as the model's fundamental compiler. Understanding how tokens are sliced not only helps you diagnose bizarre model hallucinations, but it also gives you immediately actionable tricks to trim API costs and save context windows.