Dev
Mock robots.txt Generator
A misconfigured robots.txt file can accidentally block search engines from crawling your content, or fail to block crawlers from low-value paths that clutter your index. Having a correct sample that shows the syntax — user-agent, disallow, allow, and the sitemap directive — in the right order helps when you are writing or auditing a real file. This tool generates a sample robots.txt with no inputs required. Each generated file contains a wildcard User-agent: * block with three Disallow paths (drawn from realistic sets like /admin/, /cart/, /checkout/, /private/, /search, /account/), an Allow: / to permit everything else, a Googlebot-specific block explicitly allowing all, and a Sitemap line pointing to a randomly selected example domain. Copy the output into your site's robots.txt at the web root, then replace the Sitemap URL with your real sitemap location and adjust the Disallow paths to match your site structure. A critical reminder: robots.txt is publicly visible and honours only well-behaved crawlers — it does not protect private pages from direct access or from malicious bots.
How to use
- Choose your options above
- Click Generate
- Copy your result
Detailed instructions
- Click Generate to produce a robots.txt.
- Copy it into your site root.
- Adapt the disallow rules to your site.
- Never use it to hide sensitive pages.
Use Cases
- •Learning robots.txt syntax
- •Documenting crawl rules
- •Testing a robots.txt parser
- •Seeding an example file
- •Demoing SEO configuration
Tips
- →robots.txt is a request, not security.
- →It is publicly visible.
- →Reference your sitemap in it.
- →Disallow low-value or private paths.
FAQ
what does the generated robots.txt include
A User-agent: * block with three Disallow paths, an Allow: /, a separate Googlebot block with Allow: /, and a Sitemap directive. The Disallow paths are drawn from sets like /admin/, /cart/, /checkout/, /private/, /search, and /account/, randomised each run.
does robots.txt protect private pages from being accessed
No. robots.txt only requests that well-behaved crawlers skip listed paths — it does not block HTTP access. The file is publicly visible, so listing a path can actually advertise it. Use authentication and server-side access controls to protect genuinely private content.
what paths should I typically disallow
Admin and internal areas, cart and checkout flows, search-results pages, duplicate or draft content, and other URLs with no SEO value. Disallowing these keeps them out of search indexes without affecting crawlability of your public pages. Reference your sitemap to help crawlers find what you do want indexed.
why is there a separate googlebot block in the output
User-agent-specific blocks override the wildcard block for that crawler. A Googlebot block with Allow: / explicitly tells Google to crawl everything not blocked by the wildcard section. It is a common pattern to give Google a permissive rule while keeping the wildcard block conservative.
You might also like
Popular tools from other categories that share themes with this one.
Try these next
More free tools from other corners of the catalog, picked by shared themes.