Robots.txt Generator
Robots.txt Generator builds one crawl-rules block for a single user-agent — a star, Googlebot, Bingbot or a custom name.
robots.txt only asks well-behaved crawlers not to fetch a path — it does not guarantee a page stays out of search results, and it is not a security or access-control mechanism.
How to use it
- Choose a user-agent and add Allow or Disallow rules
- Optionally set a crawl-delay or sitemap URL
- Copy the generated robots.txt and publish it at the root of your site
Add Allow and Disallow rules (each path must start with a slash), an optional Crawl-delay and an optional Sitemap line, and it assembles a plain-text file for the site root. It does not produce multiple user-agent groups in one run, and it does not check wildcard syntax: whatever you type after the slash, including star or dollar signs, is written verbatim and honoured only if the crawler supports it. Crawl-delay is emitted as given but is not universally supported and is ignored by Google. Control characters and line breaks are rejected to prevent injection. Remember that robots.txt only asks well-behaved crawlers not to fetch a path — it does not keep a page out of search results and is not access control; a listed path stays publicly readable in the file itself.
FAQ
No. Each run produces one User-agent group. Generate again with a different user-agent for a second block and combine the outputs yourself.
No. The path just has to start with a slash. Anything after that, including wildcard patterns, is written as-is and only works if the target crawler supports that syntax.
No. robots.txt asks well-behaved crawlers not to fetch a path; a URL can still be indexed from other signals. It is not security either — the file, and every path in it, is public.