Skip to content

AI-crawler robots.txt generator

Choose which AI crawlers may read your site.

Allow the crawlers that help people find you, and block the ones you would rather keep out. Get a robots.txt you can drop in as is.

Free. Runs in your browser. Nothing you type is sent anywhere.

  • Blocked
  • Allowed
  • Allowed
  • Blocked
  • Allowed
  • Allowed
  • Allowed
  • Allowed
  • Allowed
  • Allowed
  • Blocked
robots.txt
# AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Save this as robots.txt in the root of your site. robots.txt is a request that reputable crawlers honour, not a lock. Crawler names and behaviour change, so check each provider's own documentation before relying on this.

Search is not training

Several providers run separate crawlers for search, for on-demand fetching and for training. You can treat them differently.

A request, not a lock

robots.txt asks crawlers to behave. Reputable ones do. It does not secure private content.

Check the sources

Crawler names and rules change. Confirm each provider's current documentation before you rely on a file.

More tools: all free tools.

Make it stick on every release.

Omnishown checks each change to your site and sends a pull request when something like this goes missing.

  • No card, no commitment
  • Every fix arrives as a pull request you approve
  • Planned pricing from ₹2,999 per site per month
I am