Robots.txt Generator
Ready: 1 group(s), 0 rule(s).
Create a clean, valid robots.txt without memorising the syntax. Add one or more user-agent rules, list the paths to allow or disallow, drop in your sitemap URL, then copy the formatted file straight to your server root. Everything runs in your browser, so nothing you type is sent anywhere.
How to use
- Set the User-agent for your first rule. Use * to target all crawlers, or a name like Googlebot for a specific one.
- List the paths to block under Disallow, one per line (for example /admin). Add any exceptions under Allow.
- Add another user-agent rule if you need different instructions for different crawlers.
- Paste your sitemap URL in the optional Sitemap field, starting with http or https.
- Copy the generated robots.txt and save it as robots.txt in your site root.
Examples
- Block all crawlers from /admin and /private, allow /public, with a sitemap: User-agent: *, Allow: /public, Disallow: /admin, Disallow: /private, Sitemap: https://example.com/sitemap.xml
- Allow everything (the most permissive file): User-agent: * with a single empty Disallow line, which means nothing is blocked.
- Two rules: block Googlebot from /no-google, and block all other crawlers from /private.
FAQs
- Where do I put the robots.txt file?
- It must live at the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers only read it from that exact location, not from subfolders.
- What does an empty Disallow line mean?
- An empty Disallow (Disallow: with nothing after it) means nothing is blocked, so that user-agent can crawl everything. It is the standard way to allow full access.
- Does Allow override Disallow?
- For Google and Bing, the more specific rule wins, and Allow can carve out an exception inside a disallowed folder. Order does not decide it, path length and specificity do.
- Will robots.txt keep a page out of search results?
- No. Disallow stops crawling, but a blocked URL can still be indexed if other pages link to it. To keep a page out of results, use a noindex meta tag and let crawlers reach it.
- Do I need to add a sitemap line?
- It is optional but recommended. Listing your sitemap URL helps crawlers discover your pages faster. You can leave the field blank if you do not have one.
- Is the path matching case sensitive?
- Yes. Paths in robots.txt are case sensitive, so /Admin and /admin are treated as different paths. Match the exact casing your URLs use.
Related tools
- Meta Tag GeneratorGenerate clean HTML meta tags: title, description, canonical, and robots. Get a ready to paste block with length warnings built in.
- Open Graph GeneratorGenerate Open Graph and Twitter Card meta tags from a title, description, URL, image, and type. Copy a ready to paste block in seconds.
- SERP Snippet PreviewPreview how your page title and meta description may appear in search results, with live character counts and length warnings.
- Keyword Density CheckerPaste text and see the most frequent words plus their density percentage. Ignore stop words and pick how many top words to show.
- Meta Tag and OG Preview GeneratorGenerate title, description, Open Graph, and Twitter tags, then preview how your page looks in Google, Facebook, X, and LinkedIn.
- UTM BuilderBuild a campaign tracking URL with utm_source, utm_medium, utm_campaign, and optional term and content. Values are encoded for you.