Karpathy drops 3.5-hour masterclass on how LLMs actually work

If you have ever tried to figure out how modern AI actually works under the hood, you have probably run into an impenetrable wall of academic jargon.

OpenAI co-founder Andrej Karpathy just dropped a massive 3-hour-and-31-minute masterclass on YouTube titled "Deep Dive into LLMs like ChatGPT." It is basically the ultimate insider tour of how large language models are built from scratch.

Despite the runtime of a Christopher Nolan movie, Karpathy designed this lecture specifically for a general audience. You do not need a computer science degree or advanced math to follow along.

From raw text to DeepSeek-R1 reasoning

Karpathy breaks the entire LLM pipeline into three fundamental stages: pretraining, supervised finetuning (SFT), and reinforcement learning.

First comes pretraining, where raw internet data gets turned into a base model. Karpathy explains the brutal data cleaning required for datasets like FineWeb, which spans over 1.2 billion web pages, and breaks down tokenization tools like Byte Pair Encoding. For context, GPT-4 uses a vocabulary of 100,277 tokens.

He also shared a wild stat on hardware efficiency. When OpenAI trained the 1.6-billion-parameter GPT-2 in 2019, it cost around $40,000. Karpathy recently replicated that exact training run using his lightweight llm.c framework for just $672, and noted it could soon drop to around $100.

Then comes post-training. In the supervised finetuning section, Karpathy introduces what he calls "LLM Psychology." He walks through why models hallucinate, why they struggle with basic spelling or counting letters, and why "models need tokens to think." If you force a model to spit out an answer immediately without room to reason, it is basically guessing.

Finally, he tackles reinforcement learning, drawing a straight line from AlphaGo and RLHF to the recent breakthrough of DeepSeek-R1, showing how models learn complex problem-solving purely through practice and self-verification.

Why it matters

If you work with AI in any capacity, this video is arguably the most valuable 211 minutes you can spend this year.

Instead of memorizing superficial prompt engineering hacks, understanding how token prediction and working memory actually operate helps you get radically better outputs. You instantly understand why giving a model scratchpad space reduces hallucinations, and where LLM capabilities are fundamentally jagged.

For the broader tech industry, Karpathy is continuing his habit of democratizing what used to be closely guarded frontier laboratory knowledge. By laying out the blueprint from pretraining costs to DeepSeek-style reinforcement learning on simple Excalidraw whiteboards, he demystifies the aura surrounding frontier labs.

We think Karpathy's rare gift is bridging cutting-edge systems engineering with dead-simple mental models. Yes, sitting through three and a half hours requires real focus, but getting a world-class education from one of AI's chief architects without paying a dime is an unbeatable deal.