Skip to content
IFAI

Industry

Meta drops controversial AI web scraping clause

Published Reach 50

Meta just purged language from its commercial terms after critics warned it could let the company scrape news sites to train AI models without respecting robots.txt. The tech giant pushed back against the accusations, clarifying that the consolidated clause only applied to ads and business tools. Meta insists its web crawlers haven't changed and publishers can still block its bots normally.

Meta drops controversial AI web scraping clause
Image: techcrunch.com

If you run a website and keep a Meta business account for ads or social posting, your user agreements just got a little too close for comfort.

Meta has quietly removed a controversial clause from its commercial terms of service after legal experts and publishers sounded the alarm that the company was quietly giving itself permission to scrape the entire web for AI training.

The fine print that bypassed robots.txt

Earlier, Meta slipped in language stating it could use any content from a business account's domains, web pages, or URLs to provide, develop, improve, research, train, and maintain its products.

That one word, train, set off immediate red flags. Digital law professor James Grimmelmann from Cornell and Nieman Lab pointed out that this effectively gave Meta a free pass to scrape publishers' sites to train its AI models, potentially overriding any protections publishers had set up in their robots.txt files.

Meta won a major copyright lawsuit last year, but it is still facing a wave of active copyright battles alongside the rest of the industry. Sneaking broad training rights into standard ad terms looked like an end run around publisher consent.

Meta communications VP Andy Stone and spokesperson Daniel Roberts quickly pushed back, calling any claims of a crawling policy shift false. By morning, Meta had wiped the controversial language completely, clarifying that the consolidated terms only ever applied to advertising and business tools and did not give Meta any new scraping privileges.

Why it matters

For publishers and site owners, this is a clear warning that technical barriers like robots.txt are only as good as the legal terms you sign. If you operate an ad account or business page, a single hidden clause in platform terms can inadvertently license your entire web footprint.

We think Meta's maneuver looked like a classic tech reach, quietly broadening ad-scraping definitions to encompass the insatiable data hunger of foundation models. Rolling it back immediately was the only sensible PR move, especially with copyright courts breathing down its neck. But while Meta reaffirmed that robots.txt remains respected, the temptation to harvest business-linked domains has not vanished, and publishers will be reading the fine print twice as closely from now on.

Sources (2)

  1. [1]
    theverge.com(opens in a new tab)Originaltheverge.com
  2. [2]
    techcrunch.com(opens in a new tab)Reporttechcrunch.com

Similar events

Past happenings like this one, for context.

Best of AI, daily