Build rules from scratch or fetch and validate a live site's robots.txt. See exactly which named AI crawlers; training, search, and chat-fetch bots; are allowed or blocked, and flip any of them with one click.
No sitemap added yet.
Computed from the rules above. Quick actions add or remove an exact User-agent group for that bot.
GPTBot
OpenAI · training
OAI-SearchBot
OpenAI · search
ChatGPT-User
OpenAI · user-triggered fetch
ClaudeBot
Anthropic · training
Claude-SearchBot
Anthropic · search
Claude-User
Anthropic · user-triggered fetch
PerplexityBot
Perplexity · search
Perplexity-User
Perplexity · user-triggered fetch
Google-Extended
Google · training
CCBot
Common Crawl · training
Bytespider
ByteDance · training
meta-externalagent
Meta · training
Amazonbot
Amazon · training
Applebot-Extended
Apple · training
User-agent: * Disallow: /wp-admin/
robots.txt only asks well-behaved crawlers to stay out; it is not an access control or security boundary.
robots.txt at your site's root (e.g. /robots.txt). A subdomain needs its own copy.Start by checking your existing file before replacing it. Save a backup so you can restore earlier rules if needed.
Choose the crawler group you want to manage. The wildcard user agent applies broadly, while named groups target specific crawlers.
Add paths carefully, then review the preview and crawler matrix. Download the file only after checking every rule against your intended policy.
A Disallow rule for /drafts/ asks matching crawlers to avoid that directory. An Allow rule can describe a more specific permitted path.
For example, a site could restrict /drafts/ while allowing /drafts/public-example/. Check overlapping rules with the crawler documentation before publishing.
Paths are case sensitive. A small spelling change can affect which URLs match, so copy paths from real pages rather than guessing.
Robots.txt manages crawling requests. A blocked URL may still appear in search results when other pages link to it.
Use a supported noindex directive when your goal is removing a page from search. The crawler must access the page to see that directive.
Protect confidential pages with authentication or server access rules. Never publish secret paths or assume a robots rule secures a document.
Decide separately whether you want discovery in search products and use for model training. Read each operator policy before choosing a rule.
The matrix helps review named groups, but it does not prove that every bot will obey them. User-requested fetches can follow different policies.
Avoid blocking useful search access accidentally when your intention is only to restrict training. Recheck your policy when vendors change their documentation.
Place robots.txt at the root of the relevant host, such as https://example.com/robots.txt. Each subdomain needs its own configuration.
Open that address after deployment. Confirm that it returns the intended plain text rather than a login screen or an HTML error page.
Check important URLs in Search Console and monitor crawling after changes. Keep required styles and scripts accessible so search engines can render public pages.
If a live import fails, confirm that the host is public and the address is correct. You can still build rules manually.
If the file looks correct but access fails, inspect firewall rules and hosting protection. Robots.txt is only one part of the access path.
Review the file after migrations, staging launches, and major URL changes. Remove temporary restrictions before making a site public.
Common questions
Use the supplied controls to prepare your file, review its contents, and download it when ready.
Most major AI labs run a training crawler, a search/retrieval crawler for their own answer engine, and a user-triggered fetcher for pages a person explicitly asked about. They're controlled separately, and several user-triggered fetchers are documented as ignoring robots.txt by design since a person; not an autonomous crawler; requested the page.
No. It only works because well-behaved crawlers choose to respect it, and it does not hide a URL from anyone who already has the link. Use authentication or server-level access rules for anything that actually needs to stay private.
It means allow everything; the opposite of what many people assume. It's the single most common hand-written robots.txt mistake, which is why this tool flags it as a note whenever it appears.