Cloudflare's new API crawls entire sites for AI
Cloudflare just launched an open beta for its /crawl endpoint in Browser Rendering, letting developers scrape an entire website into Markdown or structured JSON with a single API call. The tool automatically obeys robots.txt and is built specifically to make gathering clean data for RAG pipelines and model training headache-free.