This robots.txt generator builds the file that tells crawlers which parts of a site they may request. Set the default access, add Disallow and Allow paths, an optional crawl delay, blocks for specific bots and a Sitemap line, then copy the result. It starts with /admin/, /api/ and /private/ as Disallow paths, which you can edit or remove, and it runs in your browser.
How to use the robots.txt generator
- Under Default Access, choose Allow all crawlers or Block all crawlers.
- Add your Sitemap URL and, if you need one, a Crawl Delay in seconds.
- Edit the Disallow Rules, using Add Disallow Path for more, and add any Allow Rules.
- Under Specific User-Agent Rules, tick the bots you want kept off the whole site.
- Press Generate robots.txt, then Copy, and save the text as a plain file named robots.txt in the root of your site.
- Open yourdomain.com/robots.txt in a browser to confirm the live file matches what you generated.
Example robots.txt file
With the default paths, GPTBot ticked and a sitemap address, the robots.txt generator writes:
# robots.txt generated by InDemand Tools
# https://indemandtools.com
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlGoogle applies the most specific matching rule, so /admin/ stays blocked even though the group starts with Allow: /. A crawler obeys only the group that names it most specifically, which is why GPTBot reads its own two lines and ignores the rest.
Crawl delay, Allow rules and Block all
- The robots.txt generator writes a Crawl-delay line only for a whole number above 0. Google does not support the rule, though some other crawlers read it.
- Block all crawlers writes
Disallow: /underUser-agent: *. Allow paths you add still open those folders to crawlers that read Allow, such as Googlebot. - The bot list covers Googlebot, Bingbot, GPTBot, AhrefsBot, SemrushBot, MJ12bot and DotBot. Tick Googlebot or Bingbot only if you want that search engine kept off the whole site.
Block AI crawlers in robots.txt
Many site owners now want to keep AI companies from training on their content. The generator has a checkbox for GPTBot, the crawler OpenAI uses for its generative AI models; OpenAI says disallowing it means the site's content should not be used in that training. Ticking it adds User-agent: GPTBot with Disallow: /.
Other AI crawlers are not in the list, so add their groups below the output by hand:
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /Google-Extended controls whether Google may use your pages for its AI models, and blocking it does not affect Google Search. The guide to robots.txt for AI crawlers lists more user agents and what each company says they do. If you want to appear in AI search answers instead, optimizing for ChatGPT search explains which OpenAI crawler to leave open.
A robots.txt file generator can only write requests. Reputable crawlers follow the file, but it cannot stop a bot that ignores it.
Why crawl budget matters
Google doesn't crawl every page of a site every day. Its crawl budget guide says this mainly matters for large sites, roughly a million pages or more, or 10,000 or more pages that change daily; for smaller sites an up-to-date sitemap is usually enough, and the XML sitemap generator builds one. On any site, robots.txt keeps crawlers away from URLs you never want crawled, such as admin screens or differently sorted copies of the same page. I always block /admin/ and /temp/, since neither holds anything a searcher needs.
Blocking a URL in the robots.txt file does not keep it out of Google's index if other pages link to it. To remove a page from results, let Google crawl it and add a noindex robots tag, which the meta tag generator can write. The rest of the SEO tools cover sitemaps, redirects and structured data.
