250 bad docs can poison an LLM

Anthropic, the UK AI Safety Institute, and The Alan Turing Institute just dropped the largest LLM poisoning study to date, and the takeaway is unsettling.

For years, AI researchers assumed that poisoning a large language model required controlling a percentage of its training data. Under that assumption, as models scale up to trillions of tokens, pulling off a backdoor attack felt practically impossible because an attacker would need millions of poisoned web pages.

It turns out that assumption was dead wrong.

Size doesn't protect models from backdoors

The researchers trained 72 models ranging from 600M to 13B parameters on varying data volumes. They tested a denial-of-service backdoor attack where inserting a specific trigger phrase, `<SUDO>`, causes the model to output high-perplexity gibberish while behaving normally on clean prompts.

Each poisoned sample was absurdly simple to make: take normal text, slap `<SUDO>` onto it, and append 400 to 900 random tokens sampled across the vocabulary.

The results were shocking: attack success depended entirely on the absolute number of malicious documents, not their share of the dataset. While 100 poisoned documents failed to reliably set a trap, injecting just 250 malicious documents succeeded consistently across all model sizes, from 600M all the way to 13B parameters.

In the 13B model run, those 250 documents amounted to about 420,000 tokens—roughly 0.00016% of the entire training corpus. Doubling the clean training tokens did nothing to dilute the backdoor.

Why it matters

This completely shatters the comfort blanket of "safety through scale."

If corrupting a foundation model requires a fixed, tiny number of documents rather than millions, data poisoning becomes a remarkably low-cost attack vector. Planting 250 blog posts or forum entries that end up in Common Crawl is well within reach of a lone bad actor, let alone a nation-state.

There are caveats, of course. The researchers only tested a basic denial-of-service attack, and it remains an open question whether complex exploits—like generating subtle software vulnerabilities or bypassing safety guardrails—can be pulled off with the same tiny sample size.

Still, we think this is a wake-up call for pretraining teams everywhere. Scrapers can no longer rely on volume to wash away bad data. Pretraining data pipelines will need to treat low-frequency, anomalous patterns as high-severity risks starting today.