Skip to main content

Robots.txt Generator

Robots.txt Generator builds one crawl-rules block for a single user-agent — a star, Googlebot, Bingbot or a custom name.

Rules
robots.txt

robots.txt only asks well-behaved crawlers not to fetch a path — it does not guarantee a page stays out of search results, and it is not a security or access-control mechanism.

How to use it

  1. Choose a user-agent and add Allow or Disallow rules
  2. Optionally set a crawl-delay or sitemap URL
  3. Copy the generated robots.txt and publish it at the root of your site

Add Allow and Disallow rules (each path must start with a slash), an optional Crawl-delay and an optional Sitemap line, and it assembles a plain-text file for the site root. It does not produce multiple user-agent groups in one run, and it does not check wildcard syntax: whatever you type after the slash, including star or dollar signs, is written verbatim and honoured only if the crawler supports it. Crawl-delay is emitted as given but is not universally supported and is ignored by Google. Control characters and line breaks are rejected to prevent injection. Remember that robots.txt only asks well-behaved crawlers not to fetch a path — it does not keep a page out of search results and is not access control; a listed path stays publicly readable in the file itself.

FAQ

We use cookies to run this site and, with your consent, to understand usage and personalize ads. You can change your choice at any time.