Robots.txt Generator
Create a valid robots.txt file: allow or block all crawlers, disallow folders, block AI training bots like GPTBot and CCBot, add your sitemap and crawl-delay, and test any URL against the rules. Free.
- Upload
- Options
- Download
- ✓ 100% free
- ✓ No sign-up
- ✓ No limits
- 🔒 Files never leave your device
How to: Robots.txt Generator
- Choose the default rule and list folders to disallow or allow.
- Optionally block AI crawlers and add your sitemap URL.
- Test a path against the rules, then download robots.txt and upload it to your site root.
Every search engine and most bots check /robots.txt before crawling a site. One misplaced slash can hide a whole website from Google, while a missing rule can let crawlers waste time on endless filter pages. FreePDF’s robots.txt generator builds a correct file from simple options, can opt your site out of the main AI-training crawlers, adds your sitemap, and includes a tester that tells you whether a specific URL would be allowed – before you upload anything.
Common uses
- Create a first robots.txt for a new website.
- Block admin, cart, search and staging folders from crawlers.
- Opt out of AI model training crawlers such as GPTBot, CCBot and Google-Extended.
- Point search engines at your XML sitemap.
- Check why a page is not being crawled by testing it against your rules.
Tips for best results
- Never disallow CSS and JavaScript folders – Google needs them to render pages.
- Use a trailing slash for folders (/admin/) so /administrator-guide is not blocked too.
- * matches any characters and $ marks the end of a URL, e.g. Disallow: /*.pdf$
- Do not use robots.txt to hide private content; protect it with a login.
- After uploading, open yoursite.com/robots.txt in a browser to confirm it is live.
How robots.txt rules are read
A file contains groups that start with one or more User-agent lines followed by Allow and Disallow rules. A crawler uses the most specific group that names it, or the * group otherwise. Within a group, Google and Bing apply the rule with the longest matching path, and Allow wins a tie. That is why “Disallow: /shop/” plus “Allow: /shop/products/” blocks the shop but keeps product pages crawlable. The built-in tester follows the same logic.
Blocking AI crawlers
Several companies crawl the web to train AI models and publish user-agent names you can block: GPTBot (OpenAI), Google-Extended (Google’s AI training control), CCBot (Common Crawl), ClaudeBot (Anthropic), PerplexityBot and Bytespider. Ticking the option adds a Disallow: / group for each. Reputable operators honour these rules, though robots.txt is voluntary and cannot technically stop a bot that ignores it. Blocking Google-Extended does not affect normal Google Search.
Crawl-delay and sitemaps
Crawl-delay asks bots to wait a number of seconds between requests. Bing and Yandex respect it; Google ignores it and adjusts speed automatically. Only use it if bots are overloading a small server. The Sitemap line can appear anywhere in the file and helps every search engine discover your XML sitemap. You can list several sitemaps, each with its own full https:// URL.
Why use FreePDF for Robots.txt Generator?
| FreePDF | Typical upload-based sites | |
|---|---|---|
| What you type is sent to a server | No – stays in your browser | Often |
| Sign-up required | Never | Often for bigger jobs |
| Hourly / daily task limits | None | Common on free plans |
| Works offline after loading | Yes, for most tools | No |
Frequently asked questions
Where does robots.txt go?
At the root of your domain, e.g. https://www.example.com/robots.txt. Each subdomain needs its own file.
Does robots.txt hide pages from Google?
It stops well-behaved crawlers from fetching pages, but a blocked URL can still be indexed if other sites link to it. Use a noindex tag to keep a page out of results.
What does “Disallow:” with nothing after it mean?
It allows everything. “Disallow: /” blocks the entire site.
Is robots.txt case-sensitive?
Paths are case-sensitive (/Admin/ differs from /admin/); user-agent names are not.
How quickly does Google notice changes?
Google usually re-fetches robots.txt within a day.
Is Robots.txt Generator really free?
Yes. Every tool on FreePDF is 100% free with no file limits, no sign-up and no watermarks. The site is supported by unobtrusive ads.
Are my files uploaded to a server?
No. FreePDF processes your files directly in your browser using JavaScript. Your documents never leave your device, so they stay completely private.
Does it work on mobile?
Yes. FreePDF works in any modern browser on Android, iPhone, iPad, Windows, Mac and Linux.