AI bots push Wikimedia's bandwidth up 50%

Wikipedia's servers were built to survive global breaking news events, but it turns out the biggest threat to their bandwidth isn't curious humans. It's automated, hungry AI bots.

The Wikimedia Foundation revealed that bandwidth demands for multimedia downloads on Wikimedia Commons have spiked by an eye-popping 50% since January 2024. And the surge is almost entirely driven by scrapers pulling images, video, and audio to train AI models.

Bots consume the most expensive traffic

The real problem isn't just the sheer volume of bot requests, but how expensive they are under the hood.

According to Wikimedia, crawler bots account for about 35% of overall pageviews, but they make up a massive 65% of the foundation's most resource-heavy traffic. The disparity comes down to human browsing habits versus machine hunger.

When humans browse, they tend to look at similar, trending topics that sit nicely in nearby cache layers. Bots, on the other hand, bulk-read everything, scraping obscure and long-forgotten files that haven't been touched in years.

Those obscure requests bypass local caches entirely and hit Wikimedia's core data center directly. It puts massive compute and cloud costs on Wikimedia, forcing their site reliability engineers to spend endless hours blocking scrapers just to keep the lights on for normal visitors.

The open web is fighting back

Wikimedia Commons hosts millions of public domain and openly licensed media files, making it ground zero for AI firms hunting for training data without copyright lawsuits.

They aren't the only ones feeling the squeeze. Open source advocate Drew DeVault recently pointed out that modern AI bots routinely ignore robots.txt protocols altogether. Pragmatic Engineer's Gergely Orosz also vented recently after scrapers from companies like Meta sent bandwidth bills through the roof on his own projects.

Tech companies are trying to offer countermeasures, like Cloudflare’s new AI Labyrinth that traps bots in infinite loops of AI-generated content. But it is fundamentally an uphill battle.

Why it matters

We think the open internet's social contract is officially breaking down.

Wikimedia operates on community donations, not venture capital. Watching multi-trillion-dollar tech companies chew up 50% more bandwidth to train proprietary commercial models—while offloading the cloud bill onto a non-profit—looks wildly unsustainable.

If non-profits and independent publishers cannot afford to keep their doors wide open to bots, they will inevitably retreat behind mandatory logins, CAPTCHAs, and paywalls.

That directly hurts everyday users who rely on the web for free, accessible knowledge. AI companies are treating the open web like an infinite open bar, but if they keep running up the tab without paying, they might end up poisoning the very well they drink from.