Dev
Mock robots.txt Generator
Used by developers, writers, and creators worldwide.
A mock robots.txt generator produces example robots.txt files for learning, testing, and documentation. The robots.txt file tells search engine crawlers which parts of a site they may and may not visit, using a simple but specific syntax of user-agent, disallow, and allow rules. This tool emits a valid robots.txt with crawl rules and a sitemap reference. Click generate and copy it into a file or a tutorial. It is ideal for learning robots.txt syntax, documenting crawl rules, and testing a parser. The file follows the real format, so it shows how to block private paths, allow the rest, and point crawlers to a sitemap. A word of caution: robots.txt is a request, not a security control — it discourages well-behaved crawlers but does not protect anything, and it is publicly visible. Adapt the rules to your own site, and never rely on robots.txt to hide sensitive pages.
Loading usage…
Free forever — no account required
How to use
- Choose your options above
- Click Generate
- Copy your result
Detailed instructions
- Click Generate to produce a robots.txt.
- Copy it into your site root.
- Adapt the disallow rules to your site.
- Never use it to hide sensitive pages.
Use Cases
- •Learning robots.txt syntax
- •Documenting crawl rules
- •Testing a robots.txt parser
- •Seeding an example file
- •Demoing SEO configuration
Tips
- →robots.txt is a request, not security.
- →It is publicly visible.
- →Reference your sitemap in it.
- →Disallow low-value or private paths.
FAQ
what does robots.txt do
It tells search engine crawlers which parts of a site they should and should not crawl, using user-agent, disallow, and allow rules. It is the first file most crawlers check, and it can also point them to your sitemap.
does robots.txt protect private pages
No. It is a request that well-behaved crawlers honour, not a security control. The file is publicly visible and ignored by bad actors, so never use it to hide sensitive pages — use proper authentication and access controls instead.
what should i disallow
Typically pages with no SEO value or that you do not want indexed — admin areas, carts, checkouts, search results, and duplicate or draft content. Allow the public content you want found, and reference your sitemap to help crawlers.