Tag

#web crawling

1 story tagged web crawling.

Cloudflare AI crawler traffic split showing 50 percent re-fetching unchanged pages, with Agent and Training categories blocked by default on ad-supported pages from September 15 2026

Cloudflare agent crawler blocks change the access deal

Cloudflare AI agent crawler rules block ad-supported pages by default from September 15. Agent builders face degraded coverage and need negotiated access, not user-agent tricks.

Lars Cornelissen · Jul 13, 2026