Robots.txt Generator
Every search engine checks your robots.txt file before crawling your site. It is a plain-text file that says which parts of your site crawlers may visit — and getting one rule wrong can hide your whole site from Google.
Fill in your rules, add your sitemap, and download a clean robots.txt — no syntax to memorise, no server access needed to draft it.
How It Works
A robots.txt file lives at the root of your site (example.com/robots.txt) and gives crawling instructions to search engines. Each block starts with a User-agent line naming which crawler the rules apply to, followed by Disallow lines for paths the crawler must skip and Allow lines for exceptions inside disallowed folders. A Sitemap line at the end points crawlers at your sitemap. Multilingual sites can also carry hreflang entries in the sitemap — a hreflang tag generator builds them.
Rules are prefix-matched: Disallow: /admin/ blocks everything under /admin/, while Disallow: / blocks the entire site — the single most dangerous line in the file, and the most common mistake. When a path matches both an Allow and a Disallow, the longer (more specific) rule wins. Crawl-delay asks polite crawlers to wait between requests; Google ignores it, but Bing and others honour it.
Worked Example
User-agent *, disallowing /admin/ and /private/, with sitemap https://example.com/sitemap.xml, produces exactly this file:
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
How to Use This Tool
Fill in your rules and click Generate Robots.txt. Copy the text or download the file, then upload it to the root of your site.
- Set the user-agent. * covers every crawler. Use Googlebot or Bingbot if you need rules for one search engine only.
- Add disallow paths. Admin areas, private folders, staging directories — anything crawlers should skip. Paths start with a slash; the tool adds one if you forget.
- Add an allow exception if needed. If one page inside a blocked folder should stay crawlable, allow it explicitly — the longer rule wins.
- Add your sitemap and crawl-delay. The sitemap line helps crawlers find your pages; crawl-delay is optional and ignored by Google. Need the sitemap file itself? An XML sitemap builder creates it from your URL list.
- Upload to your site root. The file must live at example.com/robots.txt — a robots.txt in any other folder is ignored.
After uploading, check that the file loads in a browser at yourdomain.com/robots.txt, and test it in Google Search Console's robots.txt tester before relying on it.
Tips and Practical Advice
- Never use Disallow: / on a live site. It blocks every crawler from every page. It is the number-one robots.txt accident — double-check before uploading.
- Robots.txt is a request, not a lock. Well-behaved crawlers obey it; bad bots ignore it. Never rely on robots.txt to hide sensitive content — use passwords for that.
- Do not block your CSS and JavaScript. Google needs those files to render your pages. Blocking /assets/ or similar folders can hurt your rankings.
- Keep the file small and simple. A handful of clear rules beats a long list of overlapping ones. When in doubt, allow — blocking too much costs more traffic than blocking too little.
A Complete Example File, Explained
A real robots.txt is usually short. Here is a complete, valid file for a small site, with two crawler groups and a sitemap:
# robots.txt for example.com User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /private/public-page.html Crawl-delay: 10 User-agent: Googlebot Disallow: /search/ Sitemap: https://example.com/sitemap.xml
Line by line. The first block applies to every crawler (User-agent: *): it blocks the admin area and the private folder, then makes one exception — /private/public-page.html stays crawlable, because when a path matches both an Allow and a Disallow, the longer, more specific rule wins. Crawl-delay: 10 asks polite crawlers to wait 10 seconds between requests; Google ignores it, but Bing and others honour it. The second block applies only to Googlebot and blocks the site’s internal search pages, which search engines should not index. The Sitemap line sits outside any group — it is not a rule for one crawler, it points all of them at the sitemap.
Three rules of the format are worth knowing, because they are where mistakes happen. First, matching is by prefix: Disallow: /admin/ blocks everything under /admin/, and Disallow: / blocks the entire site — while an empty Disallow: with nothing after it means the opposite, allow everything. Second, the file applies per host: www.example.com and example.com are different hosts, so each needs its own robots.txt (in practice a redirect from one to the other means only one host serves content, so only one file matters). Third, the wildcards * and $ that Google and Bing support — Disallow: /*.pdf$ to block all PDFs — are extensions, not part of the standard: RFC 9309, the specification that standardised robots.txt in 2022, defines only prefix matching, so a crawler that follows the standard strictly may not understand wildcards.
Finally, remember what the file is: a request, not a lock. Well-behaved crawlers obey it; bad bots ignore it, and a blocked URL can still appear in search results without a description if other sites link to it — robots.txt stops crawling, not indexing. To keep a page out of search results, use a noindex tag; to keep it private, use a password. Keep the file short and simple — Google reads at most the first 500 KiB — upload it to the root of your site, and test it in Search Console before relying on it.
Last reviewed: 21 November 2026.
Frequently Asked Questions
Where does the robots.txt file go?
At the root of your site: example.com/robots.txt. A robots.txt file in any other folder is ignored by crawlers. After uploading, open the URL in a browser to confirm it loads.
Does robots.txt hide pages from Google?
Not reliably. It asks crawlers not to visit a path, but Google may still list a blocked URL (without a description) if other sites link to it. To truly hide a page, use a noindex tag or a password.
What does crawl-delay do?
It asks crawlers to wait a number of seconds between requests, which helps small servers under heavy crawling. Google ignores it; Bing and several other crawlers honour it.
Can one robots.txt have rules for different crawlers?
Yes — add a block per crawler, each starting with its own User-agent line. This generator builds one block; run it again for a second crawler and paste both blocks into the same file.
Do I need a separate robots.txt for www and non-www?
Yes — robots.txt applies per host, so www.example.com and example.com each need their own file. In practice, if one host redirects to the other, crawlers follow the redirect and use the surviving host's file — but a host that serves content with no robots.txt at all is treated as fully crawlable.
What happens if a path matches both Allow and Disallow?
The longer, more specific rule wins — Allow: /private/public-page.html beats Disallow: /private/ for that one page. If both rules are the same length, Allow wins. That is how you open a single page inside a blocked folder without unblocking the folder.
Last updated: