Anti-crawling
Tarpits, cloaking, data poisoning, and crawler access control
Tarpits and labyrinths 5
Cloaking and poisoning 7
- Adversarial Cloakingaipatternbook.com
- Using AI to poison AIdropbars.be
- caddy-defender - Caddy module to block or feed garbage to AI crawlersgithub.com
- Glaze - protecting artists from generative AI style mimicryglaze.cs.uchicago.edu
- Poisoning Well - contaminating LLMs that scrape your contentheydonworks.com
- RFC 406i - The Rejection of Artificially Generated Slop406.fail
- Welcoming ObfusGit - keep your public repo out of AI training datatrustedsec.com
Access control and monetization 6
- What is Pay Per Crawl?developers.cloudflare.com
- AWS WAF announces AI traffic monetizationaws.amazon.com
- Cloudflare blocks AI agents by default now, and your agent can't telltilion.dev
- Death By Captcha - CAPTCHA solving servicedeathbycaptcha.com
- I Have Blocked OpenAI - blocking GPTBot from training on your codegavinhoward.com
- The Agentic Web Index - AI bot traffic, robots.txt compliance and spoofing statisticsknownagents.com