robots.txt

robots.txt generator & tester

Choose which search engines and AI bots may crawl your site, add your sitemap, then test any URL against the rules.

1. Crawl your website

Website URL Max pages

Pages come from your sitemap first, then from links. Crawl once – every generator below reuses the result.

2. Generate the file

Include lastmod dates changefreq priority

Google ignores changefreq and priority; lastmod is used when accurate.

Page title Group by section

Business / website name

One-line summary (shown as the quote) Extra details (optional: services, areas served, contact) Add page descriptions

WordPress: block /wp-admin/ (keep admin-ajax) Block internal search results Block cart / checkout / my-account

Other paths to block (one per line, e.g. /private/)

Search & AI bots

Bot	Used for	Rule

Sitemap URLs (one per line)

Include non-indexable URLs too

Feed title Items File name

Feed description Use all pages (no blog found / no dates)

Generated file

Test a URL

URL or path Bot

Current live robots.txt

Questions

Will blocking GPTBot remove me from ChatGPT?

GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot. Blocking GPTBot stops training use; to appear in ChatGPT search, keep OAI-SearchBot allowed.

Does robots.txt hide a page from Google?

No. robots.txt stops crawling, not indexing. To keep a page out of search results, use a noindex tag and let Google crawl it.

Where do I upload robots.txt?

It must be in the root folder so it opens at https://yourwebsite.com/robots.txt. There can be only one robots.txt per domain.